MC-GRPO: Median-Centered Group Relative Policy Optimization for Small-Rollout Reinforcement Learning

• URL: http://arxiv.org/abs/2601.22582v1
• Date: Fri, 30 Jan 2026 05:24:51 GMT
• Title: MC-GRPO: Median-Centered Group Relative Policy Optimization for Small-Rollout Reinforcement Learning
• Abstract summary: Group-relative policy optimization methods train language models by generating multiple rollouts per prompt.<n>In resource-constrained settings where the rollout budget is small, accuracy often degrades.<n>We propose Median-Centered Group Relative Policy Optimization (MC-GRPO), a simple and effective solution for small-rollout training.
• Score: 11.456510890703994
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Group-relative policy optimization methods train language models by generating multiple rollouts per prompt and normalizing rewards with a shared mean reward baseline. In resource-constrained settings where the rollout budget is small, accuracy often degrades. We find that noise in the shared baseline induces advantage sign flips, where some rollouts receive an incorrect advantage sign, and the update direction is reversed. To address this, we propose Median-Centered Group Relative Policy Optimization (MC-GRPO), a simple and effective solution for small-rollout training. Our main idea is to replace the mean baseline with a median baseline: the median is far less sensitive to outlier rewards than the mean, mitigating the sign flips under small rollout size (G). We generate one additional rollout for median reference (G+1), and compute advantages by using the group median. With an odd-sized group, exactly one completion is the median and receives zero advantage, we exclude this pivot rollout from backpropagation so the number of gradient-contributing samples per prompt remains G, preserving the core update cost of standard G-rollout training. Across various GRPO-family methods and a wide range of models and scales, this median-centered training consistently improves stability and final accuracy in the low-rollout regime, reducing the gap between G=2 and G=8 to within 1%. Code is available at https://github.com/lotusroot-kim/MC-GRPO

Related papers

• Group Adaptive Clipping Policy Optimization [11.199441103932847]
  Group relative policy optimization for reinforcement learning with verifiable rewards uses a fixed importance-sampling (IS) ratio clipping boundary across all rollouts.<n>We propose Group Adaptive Clipping Policy Optimization (GAPO), a plug-in modification to GRPO methods that adapts the clipping boundary to the rollout advantage.<n>GAPO requires no reward shaping and preserves the standard PPO/GSPO surrogate while adapting only the clipping threshold.
  arXiv  Detail & Related papers  (2026-08-31T22:32:32Z)
• Disentangling Optimization Scale from Preference Scale in DPO [0.0]
  We show that $$ entangles two distinct roles: it governs the effective inverse preference-noise scale and simultaneously rescales the optimization dynamics.<n>We propose a centered-softplus reformulation that is argmin-equivalent to DPO for $>0$, while making the inverse preference-noise-scale and learning-rate effects explicit and independently tunable.
  arXiv  Detail & Related papers  (2026-08-27T12:21:14Z)
• OR Else: A Differentiable Trust Region for Policy Optimization [2.51067108081605]
  We ask whether Output Reset (OR), a smooth one-sided saturation rule, offers a useful alternative for large language model post-training.<n>We compare PPO-clip with PPO-OR under generalized advantage estimation (GAE), and GRPO with GRPO-OR under group-relative advantages.
  arXiv  Detail & Related papers  (2026-07-20T17:07:40Z)
• MIRA: Mid-training Rubric Anchoring for Source-Aware Data Selection [59.34281845159177]
  We propose MIRA, a source-aware filtering framework based on self-anchored rubric discovery.<n>On code-oriented mid-training with 21 sources and 5 source groups, MIRA outperforms selection baselines across nine code benchmarks.
  arXiv  Detail & Related papers  (2026-05-28T17:40:40Z)
• Breaking $\ extit{Winner-Takes-All}$: Cooperative Policy Optimization Improves Diverse LLM Reasoning [53.42577591449649]
  Group Cooperative Policy Optimization shifts the training paradigm from rollout competition to team cooperation.<n>GCPO replaces independent rollout scoring with team-level credit assignment.<n>It redistributes the collective team reward to each single rollout according to its average marginal contribution to the team.
  arXiv  Detail & Related papers  (2026-05-12T03:20:24Z)
• Verifier-Free RL for LLMs via Intrinsic Gradient-Norm Reward [69.99652051809737]
  In this work, we propose Verifier-free Intrinsic Gradient-Norm Reward (VIGOR)<n>VIGOR is a simple reward that uses only the policy model itself.<n>It exhibits cross-domain transfer to code benchmarks when trained only on math data.
  arXiv  Detail & Related papers  (2026-05-11T03:15:37Z)
• Signal Reshaping for GRPO in Weak-Feedback Agentic Code Repair [36.31472731207028]
  We study signal reshaping for standard GRPO under such feedback.<n>Experiments show a clear end-to-end gain: full signal-reshaped GRPO improves strict compile-and-semantic accuracy.
  arXiv  Detail & Related papers  (2026-05-08T05:41:25Z)
• F-GRPO: Don't Let Your Policy Learn the Obvious and Forget the Rare [9.330145357231462]
  We derive the probability that updates miss rare-correct modes as a function of group size.<n>We propose a difficulty-aware advantage scaling coefficient, inspired by Focal loss, that down-weights updates on high-success prompts.
  arXiv  Detail & Related papers  (2026-02-06T14:07:30Z)
• Self-Hinting Language Models Enhance Reinforcement Learning [37.311361929798714]
  We propose self-hint aligned GRPO with privileged supervision (SAGE)<n>SAGE injects privileged hints during training to reshape the rollout distribution under the same terminal verifier reward.<n> Experiments over 6 benchmarks with 3 LLMs show that SAGE consistently outperforms GRPO.
  arXiv  Detail & Related papers  (2026-02-03T05:56:20Z)
• Group Distributionally Robust Optimization-Driven Reinforcement Learning for LLM Reasoning [45.86058898829962]
  Multi-Ad Distributionally Robust Optimization (GDRO) is an optimization-first framework that moves beyond uniform reasoning.<n>We propose two independent GDRO games for post-training: Prompt-GDRO, which employs an EMA-debiased multiplicative-weight bandit sampler to target the intensive difficulty margin and upweight persistently hard groups without frequency bias; and Rollout-GDRO, which uses a shadow-price controller to reallocate rollouts across groups, maximizing gradient variance reduction on hard tasks under a fixed mean budget (compute-neutral)<n>We validate our framework on the DAPO 14.1k dataset using Q
  arXiv  Detail & Related papers  (2026-01-27T07:10:41Z)
• DISCO Balances the Scales: Adaptive Domain- and Difficulty-Aware Reinforcement Learning on Imbalanced Data [65.09939942413651]
  We propose a principled extension to GRPO that addresses inter-group imbalance with two key innovations.<n> Domain-aware reward scaling counteracts frequency bias by reweighting optimization based on domain prevalence.<n>Difficulty-aware reward scaling leverages prompt-level self-consistency to identify and prioritize uncertain prompts that offer greater learning value.
  arXiv  Detail & Related papers  (2025-05-21T03:43:29Z)
• Kalman Filter Enhanced GRPO for Reinforcement Learning-Based Language Model Reasoning [11.157278744897427]
  Group Relative Policy Optimization ( GRPO) was proposed to compute the advantage for each output by subtracting the mean reward, as the baseline, for all outputs in the group.<n>We show that using a more adaptive advantage estimation model, KRPO can improve the stability and performance of GRPO.
  arXiv  Detail & Related papers  (2025-05-12T13:09:49Z)
• Not All Rollouts are Useful: Down-Sampling Rollouts in LLM Reinforcement Learning [55.15106182268834]
  Reinforcement learning with verifiable rewards (RLVR) has emerged as the leading approach for enhancing reasoning capabilities in large language models.<n>It faces a fundamental compute and memory asymmetry: rollout generation is embarrassingly parallel and memory-light, whereas policy updates are communication-heavy and memory-intensive.<n>We introduce PODS (Policy Optimization with Down-Sampling), which decouples rollout generation from policy updates by training only on a strategically selected subset of rollouts.
  arXiv  Detail & Related papers  (2025-04-18T17:49:55Z)
• GROS: A General Robust Aggregation Strategy [49.1574468325115]
  A new, very general, robust procedure for combining estimators in metric spaces is introduced. We show that the same (up to a constant) sub-Gaussianity is obtained if the minimization is taken over the sample. The performance of GROS is evaluated through five simulation studies.
  arXiv  Detail & Related papers  (2024-02-23T17:00:32Z)
• Revisiting Rotation Averaging: Uncertainties and Robust Losses [51.64986160468128]
  We argue that the main problem of current methods is the minimized cost function that is only weakly connected with the input data via the estimated epipolar. We propose to better model the underlying noise distributions by directly propagating the uncertainty from the point correspondences into the rotation averaging.
  arXiv  Detail & Related papers  (2023-03-09T11:51:20Z)
• KL Guided Domain Adaptation [88.19298405363452]
  Domain adaptation is an important problem and often needed for real-world applications. A common approach in the domain adaptation literature is to learn a representation of the input that has the same distributions over the source and the target domain. We show that with a probabilistic representation network, the KL term can be estimated efficiently via minibatch samples.
  arXiv  Detail & Related papers  (2021-06-14T22:24:23Z)
• Examining and Combating Spurious Features under Distribution Shift [94.31956965507085]
  We define and analyze robust and spurious representations using the information-theoretic concept of minimal sufficient statistics. We prove that even when there is only bias of the input distribution, models can still pick up spurious features from their training data. Inspired by our analysis, we demonstrate that group DRO can fail when groups do not directly account for various spurious correlations.
  arXiv  Detail & Related papers  (2021-06-14T05:39:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.