Balanced Aggregation: Understanding and Fixing Aggregation Bias in GRPO

• URL: http://arxiv.org/abs/2605.04077v1
• Date: Tue, 14 Apr 2026 09:48:46 GMT
• Title: Balanced Aggregation: Understanding and Fixing Aggregation Bias in GRPO
• Abstract summary: Reinforcement learning with verifiable rewards (RLVR) has become a central paradigm for improving reasoning and code generation in large language models.<n>Standard GRPO uses sequence aggregation, while recent work has advocated token aggregation as a better alternative.<n>We show that these two rules induce different optimization biases; token aggregation introduces sign-length coupling, while sequence aggregation implicitly downweights longer responses.
• Score: 70.38763678943648
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reinforcement learning with verifiable rewards (RLVR) has become a central paradigm for improving reasoning and code generation in large language models, and GRPO-style training is widely adopted for its simplicity and effectiveness. However, an important design choice remains underexplored: how token-level policy gradient terms are aggregated within each sampled group. Standard GRPO uses sequence aggregation, while recent work has advocated token aggregation as a better alternative. We show that these two rules induce different optimization biases: token aggregation introduces sign-length coupling, while sequence aggregation implicitly downweights longer responses through sequence-level equal weighting. To address this tension, we propose \textbf{Balanced Aggregation (BA)}, a simple drop-in replacement that computes token-level means separately within the positive and negative subsets and then combines them with sequence-count-based weights. Experiments with Qwen2.5-Math-7B and Qwen3-1.7B on DAPO-17k and Polaris, evaluated on six reasoning and coding benchmarks, show that BA consistently improves training stability and final performance over standard token and sequence aggregation. Our analysis further shows that the relative effectiveness of token and sequence aggregation is largely governed by response-length variation and the positive-negative length gap, highlighting aggregation as a critical design dimension in GRPO-style RLVR.

Related papers

• HISPO: Hierarchical Importance-Sampling Policy Optimization with Entropy-Derived Segments [3.6883184884620004]
  Reinforcement learning with verifiable rewards (RLVR) has become a central approach for improving mathematical reasoning in language models.<n>Long-form completions introduce a difficult credit-assignment problem: different parts of a solution trace may contribute unevenly to final correctness.<n>We introduce Hierarchical Importance-Sampling Policy Optimization (HISPO), a segment-level policy-optimization method that constructs rollout-time entropy-derived contiguous segments.
  arXiv  Detail & Related papers  (2026-09-14T12:29:00Z)
• STARE: Surprisal-Guided Token-Level Advantage Reweighting for Policy Entropy Stability [78.63876433996107]
  We conduct a first-order gradient analysis of token-level entropy dynamics under GRPO.<n>We propose STARE, which identifies entropy-critical token subsets via batch-internal surprisal quantiles.<n>On AIME24 and AIME25, STARE outperforms DAPO and other competitive baselines by 4%-8% in average accuracy.
  arXiv  Detail & Related papers  (2026-06-17T16:13:42Z)
• N-GRPO: Embedding-Level Neighbor Mixing for Enhanced Policy Optimization [55.14402862283128]
  We introduce N-GRPO, a novel exploration strategy integrated into the Group Relative Policy Optimization framework.<n>Rather than relying on token-level sampling or native embedding-level noise, our approach leverages Semantic Neighbor Mixing.<n>N-GRPO achieves consistent improvements over strong baselines on math reasoning benchmarks and also exhibits robust generalization capabilities on out-of-distribution tasks.
  arXiv  Detail & Related papers  (2026-06-09T12:21:27Z)
• Revisiting Reinforcement Learning with Verifiable Rewards from a Contrastive Perspective [10.958642517467721]
  We present a framework for Contrastive Sequence-level Policy Optimization in RLVR.<n>ConSPO replaces GRPO's clipped ratio-based scores with length-normalized sequence log-probabilities.<n>ConSPO consistently outperforms several strong RLVR baselines on challenging mathematical reasoning benchmarks.
  arXiv  Detail & Related papers  (2026-05-13T04:02:36Z)
• TokenRatio: Principled Token-Level Preference Optimization via Ratio Matching [20.353416189523006]
  We show how to recover token-level optimality using only standard sequence-level pairwise comparisons.<n>We introduce two instantiations: TBPO-Q, which explicitly learns a lightweight state baseline, and TBPO-A, which removes the baseline through advantage normalization.
  arXiv  Detail & Related papers  (2026-05-12T15:44:33Z)
• Rethinking Importance Sampling in LLM Policy Optimization: A Cumulative Token Perspective [22.848847562976633]
  Existing methods face a fundamental bias-variance dilemma: token-level IS ratios, as adopted by PPO and GRPO, introduce bias by ignoring prefix state distribution mismatch.<n>We propose CTPO, which combines the cumulative token IS ratio with position-adaptive clipping that scales log-space clip bounds according to the natural $sqrtt$ growth of the cumulative log-ratio.
  arXiv  Detail & Related papers  (2026-05-08T06:35:02Z)
• Too Correct to Learn: Reinforcement Learning on Saturated Reasoning Data [55.84428098924793]
  We propose Constrained Uniform Top-K Sampling (CUTS), a parameter-free decoding strategy enforcing structure-preserving exploration.<n>We integrate this into Mixed-CUTS, a training framework synergizing exploitative and exploratory rollouts to amplify intra-group advantage variance.<n> Notably, Mixed-CUTS improves Pass@1 accuracy on the challenging AIME25 benchmark by up to 15.1% over standard GRPO.
  arXiv  Detail & Related papers  (2026-04-20T16:43:28Z)
• iGRPO: Self-Feedback-Driven LLM Reasoning [88.83313431248473]
  Large Language Models (LLMs) have shown promise in solving complex mathematical problems, yet they still fall short of producing accurate and consistent solutions.<n>We introduce Iterative Group Relative Policy Optimization (iGRPO), a two-stage extension of GRPO that adds dynamic self-conditioning through model-generated drafts.<n>Under matched rollout budgets, iGRPO consistently outperforms GRPO across base models.
  arXiv  Detail & Related papers  (2026-02-09T18:45:11Z)
• Orchestrating Tokens and Sequences: Dynamic Hybrid Policy Optimization for RLVR [31.43482175098666]
  Reinforcement Learning with Verifiable Rewards (RLVR) offers a promising framework for optimizing large language models in reasoning tasks.<n>Existing RLVR algorithms focus on different granularities, and each has complementary strengths and limitations.<n>We propose Dynamic Hybrid Policy Optimization (DHPO) to bridge GRPO and GSPO within a single clipped surrogate objective.
  arXiv  Detail & Related papers  (2026-01-09T07:57:40Z)
• Anchoring Values in Temporal and Group Dimensions for Flow Matching Model Alignment [61.80228667422234]
  VGPO redefines value estimation across both temporal and group dimensions.<n>It transforms the sparse terminal reward into dense, process-aware value estimates.<n>It replaces standard group normalization with a novel process enhanced by absolute values to maintain a stable optimization signal.
  arXiv  Detail & Related papers  (2025-12-13T16:31:26Z)
• Geometric-Mean Policy Optimization [117.05113769757172]
  Group Relative Policy Optimization ( GRPO) has significantly enhanced the reasoning capability of large language models.<n> GRPO is observed to suffer from unstable policy updates when facing tokens with outlier importance-weighted rewards.<n>We propose Geometric-Mean Policy Optimization (GMPO) to improve the stability of GRPO through suppressing token reward outliers.
  arXiv  Detail & Related papers  (2025-07-28T09:54:05Z)
• GRASP: Grouped Regression with Adaptive Shrinkage Priors [2.7241418453016792]
  We introduce GRASP, a simple Bayesian framework for regression with grouped predictors.<n>NBP prior is an adaptive generalization of the horseshoe prior.<n>We show that directly controlling the tails is sufficient without requiring complex hierarchical constructions.
  arXiv  Detail & Related papers  (2025-06-22T16:35:16Z)
• What is the Alignment Objective of GRPO? [30.36318490634376]
  We present a framework that enables us to characterise the stationary policies of the GRPO algorithm.<n>The precise form of preference aggregation arises from the way the reward preference model is defined and from the penalty function.<n>We provide explicit characterisations of the aggregate preference for binary questions, for groups of size two, and in the limit of large group size.
  arXiv  Detail & Related papers  (2025-02-25T15:56:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.