Spend Your Rollouts Where It Counts: Rollout Allocation for Group-Based RL Post-Training

• URL: http://arxiv.org/abs/2605.26606v1
• Date: Tue, 26 May 2026 06:41:13 GMT
• Title: Spend Your Rollouts Where It Counts: Rollout Allocation for Group-Based RL Post-Training
• Abstract summary: Reinforcement learning is the dominant paradigm for post-training large language models.<n>Group-based policy optimization methods compute advantages from multiple rollouts per prompt.<n>Pilot-Commit is a budget-aware rollout allocation framework for group-based RL post-training.
• Score: 34.81218483901967
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reinforcement learning (RL) is the dominant paradigm for post-training large language models. However, in the online, on-policy setting, rollout generation dominates the computational cost of training. Group-based policy optimization methods compute advantages from multiple rollouts per prompt, yet they indiscriminately allocate budget to prompts with collapsed reward distributions, wasting expensive rollouts on negligible learning signals. We demonstrate that group-based updates are most effective in regimes of high reward variance. Since the policy evolves throughout training, prompt informativeness must be estimated online rather than precomputed, but exhaustively evaluating every prompt is computationally prohibitive. We introduce Pilot-Commit, a budget-aware rollout allocation framework for group-based RL post-training. Pilot-Commit decouples prompt evaluation from exploitation: a pilot stage estimates per-prompt informativeness using a fraction of the budget, and the remaining rollouts are allocated to high-leverage prompts while low-signal prompts are skipped. Across multiple math reasoning benchmarks and model scales from 1.5B to 14B parameters, Pilot-Commit matches baseline accuracy with significantly lower sampling costs, reaching target accuracy up to $1.9\times$ faster than GRPO and $4.0\times$ faster than DAPO in cumulative rollouts.

Related papers

• Not All Prompts Are Equal: Exploration-Guided Prompt Scaffolding for Multimodal Reinforcement Post-Training [15.647521419163985]
  Training prompts in online reinforcement learning (RL) differ substantially in how informative they are for the current policy.<n>We propose an exploration-guided prompt scaffolding framework that adapts the training prompt distribution dynamically throughout RL post-training of multimodal large language models (MLLMs)<n>We use a teacher model to generate scaffolded rewrites that preserve the original task intent while making subsequent training more informative.
  arXiv  Detail & Related papers  (2026-09-14T05:12:05Z)
• Learning as Reasoning Unfolds: Progressive Rollout Allocation for Efficient Reinforcement Learning [23.987185167160117]
  VarIance Guided Online Rollout allocation (VIGOR)<n>VarIance Guided Online Rollout allocation (VIGOR)<n>VarIance Guided Online Rollout allocation (VIGOR)
  arXiv  Detail & Related papers  (2026-07-24T06:04:04Z)
• Where Should RL Post-Training Compute Go? Model Size, Search, Learning, and Feedback [1.5268922363885409]
  Post-training resources are often summarized by a single total FLOP budget.<n>We introduce a FLOP-accounting framework for GRPO post-training that decomposes compute into rollout/search, policy-update/learning, and reward- or feedback-model evaluation.<n>Our results suggest that RL post-training papers should report total FLOPs together with how compute is divided among model size, search, learning, and feedback.
  arXiv  Detail & Related papers  (2026-07-15T02:34:35Z)
• TRACE: A Unified Rollout Budget Allocation Framework for Efficient Agentic Reinforcement Learning [52.67756371231985]
  Reinforcement learning with verifiable rewards (RLVR) is a promising approach for enhancing reasoning and agentic behavior in large language models.<n>We introduce Tree Rollout Allocation for Contrastive Exploration (TRACE), a unified rollout allocation framework.<n>Technically, TRACE allocates rollout budget to both prompt roots and intermediate prefixes that are most likely to yield mixed terminal rewards.
  arXiv  Detail & Related papers  (2026-06-09T17:16:03Z)
• Cross-Epoch Adaptive Rollout Optimization for RL Post-Training [5.459625245330074]
  LLM post-training often relies on reinforcement learning methods that sample multiple rollouts per prompt.<n>Most existing approaches use a fixed rollout budget for every prompt, despite large differences in the training signal different prompts provide.<n>We study adaptive rollout allocation under a fixed global budget and formulate the problem as online resource allocation with prompt-level diminishing returns.
  arXiv  Detail & Related papers  (2026-06-04T02:27:51Z)
• BASIS: Batchwise Advantage Estimation from Single-Rollout Information Sharing for LLM Reasoning [9.698479933114552]
  Existing algorithms face a tradeoff between computational efficiency and sample efficiency in value estimation and policy learning.<n>We introduce post-free critic algorithm designed to address this tradeoff.<n>Experiments demonstrate that reduces MSE in value function estimation by 69% compared to REINFORCE++.
  arXiv  Detail & Related papers  (2026-05-26T17:06:41Z)
• Your Language Model is Its Own Critic: Reinforcement Learning with Value Estimation from Actor's Internal States [14.658333957936769]
  We introduce Policy Optimization with Internal State Value Estimation.<n>It obtains a baseline at negligible cost by using the policy model's internal signals already computed during the policy forward pass.<n>A lightweight probe predicts the expected verifiable reward from the hidden states of the prompt and generated trajectory.
  arXiv  Detail & Related papers  (2026-05-08T10:49:36Z)
• Where to Spend Rollouts: Hit-Utility Optimal Rollout Allocation for Group-Based RLVR [25.35547462790362]
  Group-based policy optimization methods typically allocate a fixed number of rollouts to every prompt.<n>We introduce hit utility, the posterior probability that at least one rollout in a proposed additional allocation for a prompt will be correct.<n>We propose Hit-Utility Optimal Rollout Allocation (HORA), a learning-free rollout allocation policy.
  arXiv  Detail & Related papers  (2026-05-08T01:42:25Z)
• Near-Future Policy Optimization [51.760544033045726]
  Reinforcement learning with verifiable rewards (RLVR) has become a core post-training recipe.<n>We propose textbfNear-Future textbfPolicy textbfOptimization (textbfNPO), a simple mixed-policy scheme that learns from a policy's own near-future self.
  arXiv  Detail & Related papers  (2026-04-22T16:20:41Z)
• Unbiased Dynamic Pruning for Efficient Group-Based Policy Optimization [60.87651283510059]
  Group Relative Policy Optimization (GRPO) effectively scales LLM reasoning but incurs prohibitive computational costs.<n>We propose Dynamic Pruning Policy Optimization (DPPO), a framework that enables dynamic pruning while preserving unbiased gradient estimation.<n>To mitigate the data sparsity induced by pruning, we introduce Dense Prompt Packing, a window-based greedy strategy.
  arXiv  Detail & Related papers  (2026-03-04T14:48:53Z)
• Jackpot: Optimal Budgeted Rejection Sampling for Extreme Actor-Policy Mismatch Reinforcement Learning [31.843129392507716]
  Jackpot is a framework that integrates a unified training objective that jointly updates the policy and rollout models.<n>Our theoretical analysis shows that OBRS consistently moves the rollout distribution closer to the target distribution under a controllable acceptance budget.
  arXiv  Detail & Related papers  (2026-02-05T18:57:01Z)
• Accelerating RL for LLM Reasoning with Optimal Advantage Regression [52.0792918455501]
  We propose a novel two-stage policy optimization framework that directly approximates the optimal advantage function.<n>$A$*-PO achieves competitive performance across a wide range of mathematical reasoning benchmarks.<n>It reduces training time by up to 2$times$ and peak memory usage by over 30% compared to PPO, GRPO, and REBEL.
  arXiv  Detail & Related papers  (2025-05-27T03:58:50Z)
• Not All Rollouts are Useful: Down-Sampling Rollouts in LLM Reinforcement Learning [55.15106182268834]
  Reinforcement learning with verifiable rewards (RLVR) has emerged as the leading approach for enhancing reasoning capabilities in large language models.<n>It faces a fundamental compute and memory asymmetry: rollout generation is embarrassingly parallel and memory-light, whereas policy updates are communication-heavy and memory-intensive.<n>We introduce PODS (Policy Optimization with Down-Sampling), which decouples rollout generation from policy updates by training only on a strategically selected subset of rollouts.
  arXiv  Detail & Related papers  (2025-04-18T17:49:55Z)
• CPPO: Accelerating the Training of Group Relative Policy Optimization-Based Reasoning Models [77.16976971950785]
  This paper introduces Completion Pruning Policy Optimization (CPPO) to accelerate the training of reasoning models.<n>CPPO prunes completions with low absolute advantages, significantly reducing the number needed for gradient calculation and updates.<n> Experiments show that CPPO achieves up to $7.98times$ speedup on GSM8K and $3.48times$ on Math while preserving or even enhancing the accuracy compared to the original GRPO.
  arXiv  Detail & Related papers  (2025-03-28T11:30:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.