Prune as You Generate: Online Rollout Pruning for Faster and Better RLVR

• URL: http://arxiv.org/abs/2603.24840v1
• Date: Wed, 25 Mar 2026 22:10:36 GMT
• Title: Prune as You Generate: Online Rollout Pruning for Faster and Better RLVR
• Abstract summary: We introduce arrol (Accelerating RLVR via online Rollout Pruning), an online rollout pruning method.<n>Arrol trains a lightweight quality head on-the-fly to predict the success probability of partial rollouts and uses it to make early pruning decisions.<n>Across GRPO and DAPO on Qwen-3 and LLaMA-3.2 models, arrol improves average accuracy by +2.30 to +2.99 while achieving up to 1.7x training speedup.
• Score: 78.88799271719365
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) has significantly advanced the reasoning capabilities of Large Language Models (LLMs). However, methods such as GRPO and DAPO suffer from substantial computational cost, since they rely on sampling many rollouts for each prompt. Moreover, in RLVR the relative advantage is often sparse: many samples become nearly all-correct or all-incorrect, yielding low within-group reward variance and thus weak learning signals. In this paper, we introduce arrol (Accelerating RLVR via online Rollout Pruning), an online rollout pruning method that prunes rollouts during generation while explicitly steering the surviving ones more correctness-balanced to enhance learning signals. Specifically, arrol trains a lightweight quality head on-the-fly to predict the success probability of partial rollouts and uses it to make early pruning decisions. The learned quality head can further weigh candidates to improve inference accuracy during test-time scaling. To improve efficiency, we present a system design that prunes rollouts inside the inference engine and re-batches the remaining ones for log-probability computation and policy updates. Across GRPO and DAPO on Qwen-3 and LLaMA-3.2 models (1B-8B), arrol improves average accuracy by +2.30 to +2.99 while achieving up to 1.7x training speedup, and yielding up to +8.33 additional gains in average accuracy in test-time scaling. The code is available at https://github.com/Hsu1023/ARRoL.

Related papers

• LEEPS: Latent-Guided Explore-Exploit Prompt Sampling for Efficient RLVR in Large Language Models [21.28794880416211]
  Reinforcement learning with verifiable rewards (RLVR) improves the reasoning capabilities of large language models.<n>Existing pre-rollout methods struggle to balance exploitation and exploration.<n>We introduce LEEPS, a Latent-Guided Explore--Exploit Prompt Sampler.
  arXiv  Detail & Related papers  (2026-07-30T11:50:10Z)
• Learning as Reasoning Unfolds: Progressive Rollout Allocation for Efficient Reinforcement Learning [23.987185167160117]
  VarIance Guided Online Rollout allocation (VIGOR)<n>VarIance Guided Online Rollout allocation (VIGOR)<n>VarIance Guided Online Rollout allocation (VIGOR)
  arXiv  Detail & Related papers  (2026-07-24T06:04:04Z)
• REVES: REvision and VErification--Augmented Training for Test-Time Scaling [53.197756110943395]
  We propose a two-stage iterative framework that alternates between online data/prompt augmentation and policy optimization.<n>We observe gains of +6.5 points over the RL baseline and +4.0 points over standard multi-turn training.
  arXiv  Detail & Related papers  (2026-06-17T10:37:23Z)
• On the Direction of RLVR Updates for LLM Reasoning: Identification and Exploitation [81.29776444949539]
  We argue that the direction of updates is a more critical lens for understanding RLVR's effects.<n>Our work establishes the direction of change as a key principle for analyzing and improving RLVR.
  arXiv  Detail & Related papers  (2026-03-23T15:42:24Z)
• Prompt replay: speeding up grpo with on-policy reuse of high-signal prompts [0.0]
  Reinforcement learning with verifiable rewards (RLVR) plays a crucial role in expanding the capacities of reasoning, but GRPO-style training is dominated by expensive rollouts and wastes compute on unusable prompts.<n>We propose Prompt Replay, an overhead-free online data selection method for GRPO that reuses prompts only (not trajectories) to preserve on-policy optimization.
  arXiv  Detail & Related papers  (2026-03-22T11:39:51Z)
• Contextual Rollout Bandits for Reinforcement Learning with Verifiable Rewards [69.74686029941881]
  Reinforcement Learning with Verifiable Rewards (RLVR) is an effective paradigm for improving the reasoning capabilities of large language models.<n>We propose a unified neural scheduling framework that adaptively selects high-value rollouts throughout training.<n>Experiments on six mathematical reasoning benchmarks demonstrate consistent gains in performance and training efficiency across multiple RLVR optimization methods.
  arXiv  Detail & Related papers  (2026-02-09T10:51:58Z)
• Beyond Variance: Prompt-Efficient RLVR via Rare-Event Amplification and Bidirectional Pairing [21.946965363578087]
  Reinforcement learning with verifiable rewards (RLVR) is effective for training large language models on deterministic outcome reasoning tasks.<n>Prior work shows RLVR works with few prompts, but prompt selection is often based only on training-accuracy variance.<n>We argue that an effective minibatch should provide both (i) a reliable positive anchor and (ii) explicit negative learning signals from rare failures.
  arXiv  Detail & Related papers  (2026-02-03T12:17:25Z)
• JudgeRLVR: Judge First, Generate Second for Efficient Reasoning [20.448286296459344]
  Reinforcement Learning with Verifiable Rewards (RLVR) has become a standard paradigm for reasoning in Large Language Models.<n>In this paper, we argue that discriminative capability is a prerequisite for efficient generation.<n>We propose JudgeRLVR, a two-stage judge-then-generate paradigm.
  arXiv  Detail & Related papers  (2026-01-13T11:47:42Z)
• DLER: Doing Length pEnalty Right - Incentivizing More Intelligence per Token via Reinforcement Learning [134.03095505580276]
  Doing Length pEnalty Right (DLER) is a training recipe combining batch-wise reward normalization, higher clipping, dynamic sampling, and a simple truncation length penalty.<n>DLER achieves state-of-the-art accuracy--efficiency trade-offs, cutting output length by over 70 percent while surpassing all previous baseline accuracy.
  arXiv  Detail & Related papers  (2025-10-16T20:05:57Z)
• Beyond Binary Rewards: Training LMs to Reason About Their Uncertainty [59.97939500426759]
  This paper describes RLCR, an approach to training reasoning models that jointly improves accuracy and confidence estimation.<n>We show that across diverse datasets, RLCR substantially improves calibration with no loss in accuracy.<n>We also demonstrate that verbalized confidence can be leveraged at test time to improve accuracy and calibration.
  arXiv  Detail & Related papers  (2025-07-22T17:56:01Z)
• Kevin: Multi-Turn RL for Generating CUDA Kernels [0.0]
  We develop a flexible multi-turn RL recipe that addresses unique challenges encountered in real-world settings.<n>In our evaluation setup, Kevin shows significant gains over its base model.<n>We also study its behavior across test-time scaling axes.
  arXiv  Detail & Related papers  (2025-07-16T06:33:07Z)
• Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rollouts [28.17839625646103]
  We show that a substantial portion of computational overhead can be avoided by skipping uninformative prompts before rollout.<n>We propose GRESO, an online, lightweight pre-rollout filtering algorithm that predicts and skips uninformative prompts.<n>We show that GRESO achieves up to 2.4x wall-clock time speedup in rollout and up to 2.0x speedup in total training time without accuracy degradation.
  arXiv  Detail & Related papers  (2025-06-02T19:03:00Z)
• Putting the Value Back in RL: Better Test-Time Scaling by Unifying LLM Reasoners With Verifiers [57.95157497749428]
  We propose RL$V$ that augments any value-free'' RL method by jointly training the LLM as both a reasoner and a generative verifier.<n> RL$V$ boosts MATH accuracy by over 20% with parallel sampling and enables $8-32times$ efficient test-time compute scaling.
  arXiv  Detail & Related papers  (2025-05-07T22:41:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.