advantage computation

Advantage Computation

Concept

In Group Relative Policy Optimization (GRPO) and its variants, advantage computation refers to the normalization of per-sample rewards by group-level statistics (mean and standard deviation) to produce a learning signal for each action. Under sparse-reward regimes, where most rewards within a group are identical, standard advantage normalization can collapse to zero; GRPO with State Mutation (GRPO-SMu) addresses this by diversifying the state set and, optionally, replacing leave-one-out standard deviation with the full-population standard deviation.

First seen 9/3/2026

Last seen 9/3/2026

Evidence 2 chunks

Wiki v1

WIKI

Definition

In GRPO-based reinforcement learning fine-tuning, advantage computation converts a raw reward r(s, a_j) into a normalized advantage signal:

A^{π_{θ_t}}(s, a_j) = (r(s, a_j) − μ) / σ

READ FULL ARTICLE →

NEIGHBORHOOD

No graph connections found for this entity yet. It may appear in future ingestion runs.

explore full graph →

RELATIONSHIPS

1 connections

GRPO with State Mutation ← uses 100% 2e

GRPO-SMu modifies the advantage computation to handle sparse rewards.

LINKED ENTITIES

1 links

GRPO with State Mutation USES Extracted graph relationship

CITATIONS

5 sources

5 citations — click to expand

[1] Advantage is computed as (r − μ) / σ using group mean and standard deviation of rewards. GRPO with State Mutations: Improving LLM-Based Hardware Test Plan Generation

[2] When all rewards are identical, advantages become zero and no training signal is produced. GRPO with State Mutations: Improving LLM-Based Hardware Test Plan Generation

[3] Replacing leave-one-out σ with full-population σ amplifies the advantage for a high-reward sparse sample by more than 2×, giving 2.83 for a reward of 2.0 in Case 1. GRPO with State Mutations: Improving LLM-Based Hardware Test Plan Generation

[4] GRPO-SMu computes advantages over a diversified state set S' = {s + Δs_mutation} so that σ_{S'} captures distinct bug patterns across mutations. GRPO with State Mutations: Improving LLM-Based Hardware Test Plan Generation

[5] In roughly 30% of the first 100 training steps, only one generation achieved reward > 1 while others had reward 1, characterizing the sparse-reward regime that motivates modified advantage computation. GRPO with State Mutations: Improving LLM-Based Hardware Test Plan Generation