Gradient Starvation in Binary-Reward GRPO: Why Group-Mean Centering Fails and Why the Simplest Fix Works

• URL: http://arxiv.org/abs/2605.07689v1
• Date: Fri, 08 May 2026 12:58:06 GMT
• Title: Gradient Starvation in Binary-Reward GRPO: Why Group-Mean Centering Fails and Why the Simplest Fix Works
• Abstract summary: Group Relative Policy Optimization (GRPO) is a standard algorithm for reinforcement learning from verifiable rewards.<n>We prove that the true degeneracy rate always exceeds the i.i.d. Bernoulli prediction by Jensen's inequality.<n>We show that the fixed-reference Sign advantage, $A=2r-1$, performs pass@$G$ failure descent by increasing the probability that at least one sample in the group succeeds.
• Score: 19.92945464084119
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Group Relative Policy Optimization (GRPO) is a standard algorithm for reinforcement learning from verifiable rewards, but its group-mean-centered advantage can fail under binary rewards. The failure mode is gradient starvation: when every response in a group is correct or every response is wrong, the centered advantage is exactly zero and the policy receives no learning signal. We prove that the true degeneracy rate always exceeds the i.i.d. Bernoulli prediction by Jensen's inequality, and observe a 0.69 degeneracy rate at group size four in logged Qwen3.5-9B GSM8K training. We then show that the fixed-reference Sign advantage, $A=2r-1$, performs pass@$G$ failure descent by increasing the probability that at least one sample in the group succeeds. On the full GSM8K test set across seven seeds, Sign reaches 73.8% accuracy versus 28.4% for standard normalized group-mean DrGRPO at group size four, a 45.4 point gain with $p<0.0001$. The effect is directionally consistent on Llama-3.1-8B and positive but underpowered on a MATH-500 transfer check. Pass@$k$ analysis indicates that the main benefit is search compression rather than large capacity expansion, aligning the empirical gains with recent RLVR ceiling observations.

Related papers

• Distill Skills into Weights, Not Prompts: Abstract Skills as Privileged Signals for On-Policy Self-Distillation [7.296659715681798]
  Reinforcement learning with verifiable rewards yields no group-relative signal when rollout groups are uniformly correct or uniformly wrong.<n>We propose SKALD (Skill-Anchored Latent Distillation), an on-policy self-distillation framework that uses two context views of the same Qwen3-Base model.
  arXiv  Detail & Related papers  (2026-08-10T16:43:18Z)
• Distill Where You Fail: Recovering Learning Signals of Negative RL-Groups from Adaptive Teacher Guidance [50.77757355236912]
  We propose RSTG (Recovering Learning Signals via Adaptive Teacher Guidance), which applies distillation selectively and precisely where it matters most.<n> Experiments demonstrate that RSTG substantially outperforms naive GRPO+OPD by +4.02% on math and+3.05% on code.
  arXiv  Detail & Related papers  (2026-08-01T17:29:14Z)
• The Dark Room in the Reward Channel: Dense Prediction Rewards Collapse GRPO-Trained LLM Agents -- and What Actually Works [3.040305634723913]
  We show that under group-normalized RL, a potential-based prediction reward drives every run into a degenerate absorbing state.<n>A single-episode ablation ablations the cause -- removing only GRPO's normalization turns the same reward from catastrophic into baseline parity.<n>Our central insight generalizes this: what z-scoring amplifies is a dense signal's within-group variance while all-fail groups dominate.
  arXiv  Detail & Related papers  (2026-07-23T12:50:18Z)
• CIGPO: Contextual Information-Gain Policy Optimization for Multi-Turn Evidence-Reading LLM Agents [1.4298580363875282]
  Training multi-turn evidence-reading agents with outcome-only reinforcement learning is unstable because intermediate turns receive little direct credit.<n>In HotpotQA experiments with Qwen2.5-3B-Instruct, GRPO initially improves (standard F1 0.430) but subsequently collapses to 100% format-violating outputs.<n>We propose a variance-textual strategy: by assigning per-turn rewards to intermediate evidence-reading turns, we prevent the group reward distribution from collapsing to a single value.
  arXiv  Detail & Related papers  (2026-06-26T11:50:29Z)
• Right Makes Might: Aligning Verified Hidden States Empowers RL Reasoning [55.264863369127774]
  Current methods reduce each correct rollout to a single reward bit, ignoring the geometric structure shared among their hidden states.<n>We propose Hidden-Align, an auxiliary loss function that aligns the last-layer hidden states of correct rollouts at the anchor token during RL training, with zero overhead in both training and inference.<n>On eight mathematical reasoning benchmarks, Hidden-Align improves average pass@1 over the DAPO baseline by 3.8, 6.2, and 5.4 percentage points on Qwen3-1.7B, 4B, and 14B respectively, with consistent pass@k gains across all three scales.
  arXiv  Detail & Related papers  (2026-06-02T06:51:15Z)
• VeriGate: Verifier-Gated Step-Level Supervision for GRPO [51.26100506256885]
  Group Relative Policy Optimization is an effective recipe for training reasoning models with verifier-based outcome rewards.<n>We present VeriGate, a verifier-gated extension of GRPO that addresses these limitations with three design choices.<n>We show that VeriGate improves average accuracy by about 20% and 12% for 1.5B and 7B models respectively.
  arXiv  Detail & Related papers  (2026-05-28T18:20:32Z)
• Verifier-Free RL for LLMs via Intrinsic Gradient-Norm Reward [69.99652051809737]
  In this work, we propose Verifier-free Intrinsic Gradient-Norm Reward (VIGOR)<n>VIGOR is a simple reward that uses only the policy model itself.<n>It exhibits cross-domain transfer to code benchmarks when trained only on math data.
  arXiv  Detail & Related papers  (2026-05-11T03:15:37Z)
• Learning to Hint for Reinforcement Learning [51.46328710610512]
  Group Relative Policy Optimization (GRPO) is widely used for reinforcement learning with verifiable rewards.<n> GRPO often suffers from advantage collapse when all rollouts in a group receive the same reward.<n>We propose Hint Learning for Reinforcement Learning (HiLL), a framework that jointly trains a hinter policy and a reasoner policy.
  arXiv  Detail & Related papers  (2026-04-01T09:58:08Z)
• FIPO: Eliciting Deep Reasoning with Future-KL Influenced Policy Optimization [84.58281577727566]
  We present Future-KL Influenced Policy Optimization (FIPO), a reinforcement learning algorithm designed to overcome reasoning bottlenecks in large language models.<n>FIPO addresses this by incorporating discounted future-KL divergence into the policy update, creating a dense advantage formulation that re-weights tokens based on their influence on subsequent trajectory behavior.<n> evaluated on Qwen2.5-32B, FIPO extends the average chain-of-thought length from roughly 4,000 to over 10,000 tokens and increases AIME 2024 Pass@1 accuracy from 50.0% to a peak of 58.0%.
  arXiv  Detail & Related papers  (2026-03-20T10:24:50Z)
• F-GRPO: Don't Let Your Policy Learn the Obvious and Forget the Rare [9.330145357231462]
  We derive the probability that updates miss rare-correct modes as a function of group size.<n>We propose a difficulty-aware advantage scaling coefficient, inspired by Focal loss, that down-weights updates on high-success prompts.
  arXiv  Detail & Related papers  (2026-02-06T14:07:30Z)
• Beyond Variance: Prompt-Efficient RLVR via Rare-Event Amplification and Bidirectional Pairing [21.946965363578087]
  Reinforcement learning with verifiable rewards (RLVR) is effective for training large language models on deterministic outcome reasoning tasks.<n>Prior work shows RLVR works with few prompts, but prompt selection is often based only on training-accuracy variance.<n>We argue that an effective minibatch should provide both (i) a reliable positive anchor and (ii) explicit negative learning signals from rare failures.
  arXiv  Detail & Related papers  (2026-02-03T12:17:25Z)
• Transform-Augmented GRPO Improves Pass@k [50.3707071191733]
  Group Relative Policy Optimization (GRPO) was designed to improve reasoning, but it worsens this situation through two failure modes.<n>We propose TA-GRPO (Transform-Augmented GRPO), which generates semantically equivalent transformed variants of each question.<n>This pooled computation ensures mixed rewards even when the original question is too easy or too hard, while training on diverse phrasings promotes multiple solution strategies.
  arXiv  Detail & Related papers  (2026-01-30T02:43:29Z)
• CARE What Fails: Contrastive Anchored-REflection for Verifiable Multimodal [84.71254539482369]
  Group-relative reinforcement learning with verifiable rewards (RLVR) often wastes the most informative data it already has the failures.<n>We present CARE, a failure-centric post-training framework for multimodal reasoning that turns errors into supervision.<n> CARE improves accuracy and training smoothness while explicitly increasing the share of learning signal that comes from failures.
  arXiv  Detail & Related papers  (2025-12-22T16:34:21Z)
• Don't Waste Mistakes: Leveraging Negative RL-Groups via Confidence Reweighting [24.822152032771736]
  Reinforcement learning with verifiable rewards (RLVR) has become a standard recipe for improving large language models.<n>We show that negative groups can be leveraged without extra supervision.
  arXiv  Detail & Related papers  (2025-10-09T18:01:44Z)
• C$^2$GSPG: Confidence-calibrated Group Sequence Policy Gradient towards Self-aware Reasoning [54.705168477975384]
  Group Sequence Policy Gradient (GSPG) framework for learning reasoning models.<n>C$2$GSPG simultaneously enhances reasoning performance while suppressing overconfidence.
  arXiv  Detail & Related papers  (2025-09-27T05:24:51Z)
• NGRPO: Negative-enhanced Group Relative Policy Optimization [8.641009168869195]
  A representative RLVR algorithm, GRPO, suffers from a critical limitation when all responses within a group are either entirely correct or entirely incorrect.<n>This is particularly problematic for homogeneously incorrect groups, where GRPO's advantage function yields a value of zero.<n>We propose NGRPO, an algorithm designed to convert homogeneous errors into robust learning signals.
  arXiv  Detail & Related papers  (2025-09-23T09:38:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.