Rethinking Groups in Critic-Free RLVR

• URL: http://arxiv.org/abs/2606.17250v1
• Date: Mon, 15 Jun 2026 19:49:42 GMT
• Title: Rethinking Groups in Critic-Free RLVR
• Abstract summary: Reinforcement learning (RL) has become a central paradigm for post-training large language models.<n>Existing critic-free RL methods typically generate a group of rollouts for the same question.<n>We show that its underlying function is not merely to estimate baselines but to prevent false penalties on negative samples.
• Score: 22.22297138402367
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Reinforcement learning (RL) has become a central paradigm for post-training large language models. Existing critic-free RL methods typically generate a group of rollouts for the same question to estimate value baselines for advantage computation. However, this design suffers from data inefficiency, group synchronization barriers, and inflexibility with structured rollouts. In this work, we revisit the role of the ``group'' and show that its underlying function is not merely to estimate baselines but to prevent false penalties on negative samples. Building on this insight, we propose negative token filtering, a simple and effective strategy that enables stable single-rollout training. We apply it to two batch-level advantage methods, achieving comparable performance on reasoning tasks and stronger performance on agentic tasks relative to group-based RL techniques.

Related papers

• Le Critique: Privileged Value Functions for LLM Reinforcement Learning [19.187738387494438]
  Reinforcement learning algorithms for Large Language Models (LLMs) are largely distinguished by their variance reduction strategy.<n>Group-relative methods like GRPO reduce gradient variance by sampling multiple rollouts per prompt, but provide only sequence-level credit.<n>We propose two complementary strategies to improve the performance of value function RL.
  arXiv  Detail & Related papers  (2026-08-17T15:49:36Z)
• BV-Blend: Uncertainty-Weighted Historical Baselines for Stable Critic-Free RL with Verifiable Rewards [17.873016968298483]
  Critic-free reinforcement learning with verifiable rewards (RLVR) avoids training a value function (critic) and reduces memory and compute overhead.<n> GRPO-style advantage estimation depends on prompt-local (within-prompt-group) reward statistics and can be unstable.<n>We introduce BV-Blend, a critic-free framework that stabilizes advantage estimation by combining prompt-local on-policy statistics with semantic-cluster-conditioned historical moments.
  arXiv  Detail & Related papers  (2026-06-27T03:25:53Z)
• RLVR without Ineffective Samples: Group Prioritized Off-Policy Optimization for LLM Reasoning [49.04912820721943]
  Group Prioritized Off-Policy Optimization (POPO) is a framework that exploits effective training batches without additional rollout overhead.<n>POPO comprises two key components: prioritized group replay and decoupled off-policy optimization.<n>POPO substantially accelerates RL finetuning and achieves strong reasoning performance with significantly fewer rollouts.
  arXiv  Detail & Related papers  (2026-05-31T15:06:38Z)
• BASIS: Batchwise Advantage Estimation from Single-Rollout Information Sharing for LLM Reasoning [9.698479933114552]
  Existing algorithms face a tradeoff between computational efficiency and sample efficiency in value estimation and policy learning.<n>We introduce post-free critic algorithm designed to address this tradeoff.<n>Experiments demonstrate that reduces MSE in value function estimation by 69% compared to REINFORCE++.
  arXiv  Detail & Related papers  (2026-05-26T17:06:41Z)
• Contextual Rollout Bandits for Reinforcement Learning with Verifiable Rewards [69.74686029941881]
  Reinforcement Learning with Verifiable Rewards (RLVR) is an effective paradigm for improving the reasoning capabilities of large language models.<n>We propose a unified neural scheduling framework that adaptively selects high-value rollouts throughout training.<n>Experiments on six mathematical reasoning benchmarks demonstrate consistent gains in performance and training efficiency across multiple RLVR optimization methods.
  arXiv  Detail & Related papers  (2026-02-09T10:51:58Z)
• From Absolute to Relative: Rethinking Reward Shaping in Group-Based Reinforcement Learning [7.6602542594279335]
  We propose Reinforcement Learning with Relative Rewards to shift reward shaping from absolute scoring to relative ranking.<n>We show that RLRR yields consistent performance improvements over standard group-based baselines across reasoning benchmarks and open-ended generation tasks.
  arXiv  Detail & Related papers  (2026-01-30T15:07:06Z)
• Your Group-Relative Advantage Is Biased [74.57406620907797]
  Group-based learning methods rely on group-relative advantage estimation to avoid learned critics.<n>In this work, we uncover a fundamental issue of group-based RL: the group-relative advantage estimator is inherently biased relative to the true (expected) advantage.<n>We propose History-Aware Adaptive Difficulty Weighting (HA-DW), an adaptive reweighting scheme that adjusts advantage estimates based on an evolving difficulty anchor and training dynamics.
  arXiv  Detail & Related papers  (2026-01-13T13:03:15Z)
• Stable and Efficient Single-Rollout RL for Multimodal Reasoning [66.53652874617217]
  $textbfMSSR$ (Multimodal Stabilized Single-Rollout) is a group-free RLVR framework that achieves both stable optimization and effective multimodal reasoning performance.<n>In in-distribution evaluations, MSSR demonstrates superior training compute efficiency, achieving similar validation accuracy to the group-based baseline with half the training steps.
  arXiv  Detail & Related papers  (2025-12-20T05:07:53Z)
• A Minimalist Approach to LLM Reasoning: from Rejection Sampling to Reinforce [68.99924691391048]
  We revisit GRPO from a reinforce-like algorithm perspective and analyze its core components.<n>We find that a simple rejection sampling baseline, RAFT, yields competitive performance than GRPO and PPO.<n>Motivated by this insight, we propose Reinforce-Rej, a minimal extension of policy gradient that filters both entirely incorrect and entirely correct samples.
  arXiv  Detail & Related papers  (2025-04-15T16:15:02Z)
• Take One Gram of Neural Features, Get Enhanced Group Robustness [23.541213868620837]
  Predictive performance of machine learning models trained with empirical risk minimization can degrade considerably under distribution shifts. We propose to partition the training dataset into groups based on Gram matrices of features extracted by an identification'' model. Our approach not only improves group robustness over ERM but also outperforms all recent baselines.
  arXiv  Detail & Related papers  (2022-08-26T12:34:55Z)
• Examining and Combating Spurious Features under Distribution Shift [94.31956965507085]
  We define and analyze robust and spurious representations using the information-theoretic concept of minimal sufficient statistics. We prove that even when there is only bias of the input distribution, models can still pick up spurious features from their training data. Inspired by our analysis, we demonstrate that group DRO can fail when groups do not directly account for various spurious correlations.
  arXiv  Detail & Related papers  (2021-06-14T05:39:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.