ThinkPrior: Zero-Rollout Difficulty Priors for Cold-Start Prompt Selection in RLVR

• URL: http://arxiv.org/abs/2609.09075v2
• Date: Fri, 11 Sep 2026 19:18:34 GMT
• Title: ThinkPrior: Zero-Rollout Difficulty Priors for Cold-Start Prompt Selection in RLVR
• Abstract summary: Group-relative advantages are identically zero if all rollouts in a group are correct or all are wrong.<n>History-based prompt selection must first spend target-policy rollouts to estimate difficulty.<n>ThinkPrior selects by expected learnability and then updates from training outcomes, changing neither the loss nor the gradient.<n>The measured ThinkPrior+DAPO composition reduces generated rollouts by 10.6% while both arms retain the same 3840-rollout update budget.
• Score: 0.38452047300097886
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In reinforcement learning with verifiable rewards (RLVR) trained with group relative policy optimization (GRPO), the KL-free reward-advantage term studied here depends on within-group reward variation. If all rollouts in a group are correct or all are wrong, their group-relative advantages are identically zero; these zero-advantage silent groups provide no reward-advantage gradient, yet uniform sampling spends 39% of a run's rollouts on them. History-based prompt selection must first spend target-policy rollouts to estimate difficulty, creating a cold start with rollout waste; ThinkPrior instead uses an external anchor in one offline pass to construct a zero-rollout difficulty prior before the first target-policy rollout. The verifier-scored anchor pass rate supplies an external-anchor initialization for a Beta posterior; ThinkPrior selects by expected learnability and then updates from training outcomes, changing neither the loss nor the optimizer. On Qwen2.5-Math-7B across sixteen seeds, ThinkPrior more than halves early silent groups and cuts wasted rollouts through step 30 by nearly a fifth, while we detect no difference in final accuracy. On this 250-prompt pool the fixed-budget result is a reallocation rather than a net saving. The measured ThinkPrior+DAPO composition reduces generated rollouts by 10.6% while both arms retain the same 3840-rollout update budget. The prior requires no target-policy rollout before the first selection, but the posterior thereafter uses target-policy outcomes.

Related papers

• Learn What's Left, Not What's Mastered: Saturation Aware Advantage Reweighting for Multi-Reward Policy Optimization [60.89431018071735]
  Reinforcement learning with group-relative advantages has become the de facto standard for post-training language model reasoners.<n>We show that rollouts with distinct reward profiles can receive identical advantages, and all objectives are optimized with fixed relative weights regardless of their current level of saturation.<n>We introduce saturation-aware reweighting, which standardizes each reward objective independently and adaptively discounts its contribution according to a batch-level estimate of objective saturation.
  arXiv  Detail & Related papers  (2026-08-17T04:07:50Z)
• PAIR: Pairwise-Aware Inclusion Reweighting for Adaptive Rollout Allocation in RLVR [0.0]
  Reinforcement learning with verifiable rewards spends most of its compute generating groups of long reasoning trajectories.<n>Recent allocators reduce this cost by assigning budgets to prompts, rollouts, or tokens according to a pointwise notion of difficulty or utility.<n>We identify a statistical mismatch: the unclipped leave-one-out group-relative score gradient is not a sum of independent point contributions, but a second-order U-statistic over pairs of rollouts.
  arXiv  Detail & Related papers  (2026-08-11T19:24:59Z)
• Early Verdicts, Better Budgets: Sequential Adaptive Rollout Allocation for Compute-Efficient RLVR [0.0]
  We show that a group's effectiveness is usually decided early, within the first few of its rollouts, so spending a full group on an already-decided prompt is wasteful.<n>We apply a two-threshold, SPRT-style rule that commits effective groups, abandons saturated ones after a short probe, and reallocates the freed budget to fresh prompts.<n>We prove abandonment reliability, expected rollout savings, fixed-budget yield dominance, and a link between effective-group yield and the GRPO norm.
  arXiv  Detail & Related papers  (2026-07-28T20:43:23Z)
• CIGPO: Contextual Information-Gain Policy Optimization for Multi-Turn Evidence-Reading LLM Agents [1.4298580363875282]
  Training multi-turn evidence-reading agents with outcome-only reinforcement learning is unstable because intermediate turns receive little direct credit.<n>In HotpotQA experiments with Qwen2.5-3B-Instruct, GRPO initially improves (standard F1 0.430) but subsequently collapses to 100% format-violating outputs.<n>We propose a variance-textual strategy: by assigning per-turn rewards to intermediate evidence-reading turns, we prevent the group reward distribution from collapsing to a single value.
  arXiv  Detail & Related papers  (2026-06-26T11:50:29Z)
• GD$^2$PO: Mitigating Multi-Reward Conflicts via Group-Dynamic reward-Decoupled Policy Optimization [54.596224644751565]
  Post-training reinforcement learning (RL) increasingly relies on multi-dimensional rewards to cultivate comprehensive capabilities.<n>To address this, existing methods such as Group reward-Decoupled Policy Optimization (GDPO) decompose the overall score into independent reward groups, then compute the RL loss separately within each group.<n>We propose Group-Dynamic reward-Decoupled Policy Optimization (GD$2$PO), which employs a conflict-aware filtering mechanism to mask out rollouts suffering from severe reward-wise disagreement.
  arXiv  Detail & Related papers  (2026-06-15T14:19:28Z)
• TRACE: A Unified Rollout Budget Allocation Framework for Efficient Agentic Reinforcement Learning [52.67756371231985]
  Reinforcement learning with verifiable rewards (RLVR) is a promising approach for enhancing reasoning and agentic behavior in large language models.<n>We introduce Tree Rollout Allocation for Contrastive Exploration (TRACE), a unified rollout allocation framework.<n>Technically, TRACE allocates rollout budget to both prompt roots and intermediate prefixes that are most likely to yield mixed terminal rewards.
  arXiv  Detail & Related papers  (2026-06-09T17:16:03Z)
• Breaking $\textit{Winner-Takes-All}$: Cooperative Policy Optimization Improves Diverse LLM Reasoning [53.42577591449649]
  Group Cooperative Policy Optimization shifts the training paradigm from rollout competition to team cooperation.<n>GCPO replaces independent rollout scoring with team-level credit assignment.<n>It redistributes the collective team reward to each single rollout according to its average marginal contribution to the team.
  arXiv  Detail & Related papers  (2026-05-12T03:20:24Z)
• Where to Spend Rollouts: Hit-Utility Optimal Rollout Allocation for Group-Based RLVR [25.35547462790362]
  Group-based policy optimization methods typically allocate a fixed number of rollouts to every prompt.<n>We introduce hit utility, the posterior probability that at least one rollout in a proposed additional allocation for a prompt will be correct.<n>We propose Hit-Utility Optimal Rollout Allocation (HORA), a learning-free rollout allocation policy.
  arXiv  Detail & Related papers  (2026-05-08T01:42:25Z)
• Selective Rollout: Mid-Trajectory Termination for Multi-Sample Agent RL [6.435733307123974]
  Group-relative RL training (GRPO) samples a small group of parallel rollouts for every training prompt.<n>Uses their within-group reward spread to compute per-trajectory advantages.<n>When every rollout of a prompt ends with the same reward, the group has zero reward variance and contributes no gradient.
  arXiv  Detail & Related papers  (2026-05-07T07:41:09Z)
• Learning to Hint for Reinforcement Learning [51.46328710610512]
  Group Relative Policy Optimization (GRPO) is widely used for reinforcement learning with verifiable rewards.<n> GRPO often suffers from advantage collapse when all rollouts in a group receive the same reward.<n>We propose Hint Learning for Reinforcement Learning (HiLL), a framework that jointly trains a hinter policy and a reasoner policy.
  arXiv  Detail & Related papers  (2026-04-01T09:58:08Z)
• Not All Rollouts are Useful: Down-Sampling Rollouts in LLM Reinforcement Learning [55.15106182268834]
  Reinforcement learning with verifiable rewards (RLVR) has emerged as the leading approach for enhancing reasoning capabilities in large language models.<n>It faces a fundamental compute and memory asymmetry: rollout generation is embarrassingly parallel and memory-light, whereas policy updates are communication-heavy and memory-intensive.<n>We introduce PODS (Policy Optimization with Down-Sampling), which decouples rollout generation from policy updates by training only on a strategically selected subset of rollouts.
  arXiv  Detail & Related papers  (2025-04-18T17:49:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.