SpecRoll: Fast-Slow Verifier-Feedback Adaptation for Speculative Reinforcement Learning Rollouts

• URL: http://arxiv.org/abs/2608.04962v1
• Date: Wed, 05 Aug 2026 15:32:27 GMT
• Title: SpecRoll: Fast-Slow Verifier-Feedback Adaptation for Speculative Reinforcement Learning Rollouts
• Abstract summary: We introduce SpecRoll, a speculative rollout engine that preserves the target model's sampling distribution while adapting at two timescales.<n> Lightweight future-token heads generate parallel proposals, while a Reflex module uses delayed verifier feedback to performBounded, trajectory-local hidden-state corrections.<n>Across five models ranging from 1.5B to 14B, SpecRoll 1.26-2.15x generation speedup and 1.21-2.04x end-to-end speedup over vanilla GRPO.
• Score: 3.729816445527413
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reinforcement learning (RL) post-training improves the reasoning capabilities of large language models, but autoregressive rollout generation remains a major efficiency bottleneck. Speculative decoding can accelerate generation, yet applying it during RL is difficult because the target policy continually evolves: static proposers become stale, while frequent drafter updates add substantial overhead. We introduce SpecRoll, a speculative rollout engine that preserves the target model's sampling distribution while adapting at two timescales. Lightweight future-token heads generate parallel proposals, while our proposed Reflex module uses delayed verifier feedback to perform bounded, trajectory-local hidden-state corrections without backpropagation. A complementary slow path updates the head parameters only when sustained degradation is detected. SpecRoll combines these mechanisms with concurrency-aware sparse-tree verification and exact target verification, leaving the target rollout distribution and GRPO objective unchanged. Across five models ranging from 1.5B to 14B and three mathematical reasoning datasets, SpecRoll achieves 1.26-2.15x generation speedup and 1.21-2.04x end-to-end speedup over vanilla GRPO. It also outperforms FastGRPO in both generation and end-to-end time across all 15 matched settings, with an average pairwise end-to-end gain of 1.18x. Controlled ablations show that the fast and slow adaptation paths provide complementary benefits. Our source code is available at https://anonymous.4open.science/r/SpecRoll-26062006.

Related papers

• WAR: Workload-Aware Rollouts for Synchronous Agentic Reinforcement Learning [0.0]
  WAR is a workload-aware rollout system that substantially accelerates synchronous agentic RL training.<n>Under low load, WAR enables model-free speculative decoding with SuffixDecoding.<n>Under high load, WAR shifts the optimization focus to cache-aware scheduling.
  arXiv  Detail & Related papers  (2026-07-19T15:31:13Z)
• EfficientRollout: System-Aware Self-Speculative Decoding for RL Rollouts [20.460789503141466]
  We present EfficientRollout, a system-aware self-SD framework designed to address this gap for RL rollouts.<n> EfficientRollout reduces rollout and end-to-end latency by up to 19.6% and 12.7%, respectively, over an accelerated AR rollout baseline.
  arXiv  Detail & Related papers  (2026-06-17T11:51:06Z)
• Accelerating RL Post-Training Rollouts via System-Integrated Speculative Decoding [8.43502622452414]
  We study speculative decoding as an acceleration primitive for RL rollouts.<n>We implement speculative decoding in NeMo-RL with a vLLM backend.<n>We project that combining speculative decoding with asynchronous RL yields up to 2.5x end-to-end training speedup at 235B scale.
  arXiv  Detail & Related papers  (2026-04-29T15:11:48Z)
• When RL Meets Adaptive Speculative Training: A Unified Training-Serving System [71.98182665273575]
  We present Aurora, a unified training-serving system that closes the loop by continuously learning a speculator directly from live inference traces.<n>Our design integrates an SGLang-based inference server with an asynchronous training server, enabling hot-swapped speculator updates without service interruption.
  arXiv  Detail & Related papers  (2026-02-06T18:28:54Z)
• Taming the Long-Tail: Efficient Reasoning RL Training with Adaptive Drafter [52.111923076688505]
  Training Large Language Models (LLMs) with strong reasoning capabilities marks a significant milestone, unlocking new frontiers in complex problem-solving.<n>We propose TLT, a system that accelerates reasoning RL training losslessly by integrating adaptive speculative decoding.
  arXiv  Detail & Related papers  (2025-11-20T18:59:25Z)
• Beat the long tail: Distribution-Aware Speculative Decoding for RL Training [75.75462952580796]
  We propose a Distribution Aware Speculative decoding framework that accelerates RL rollouts without altering model outputs.<n>Experiments on math and code reasoning tasks show that DAS reduces rollout time up to 50% while preserving identical training curves.
  arXiv  Detail & Related papers  (2025-11-17T19:02:12Z)
• Saber: An Efficient Sampling with Adaptive Acceleration and Backtracking Enhanced Remasking for Diffusion Language Model [98.35868970993232]
  Diffusion language models (DLMs) are emerging as a powerful and promising alternative to the dominant autoregressive paradigm.<n>We introduce efficient Sampling with Adaptive acceleration and Backtracking Enhanced Remasking (i.e., Saber) to achieve better inference speed and output quality in code generation.
  arXiv  Detail & Related papers  (2025-10-20T23:38:12Z)
• Mirror Speculative Decoding: Breaking the Serial Barrier in LLM Inference [11.957170239588535]
  Speculative decoding accelerates inference by using a draft model to look ahead.<n>Prior methods partially reduce draft cost but either degrade acceptance or introduce overheads that limit scaling.<n>We present Mirror Speculative Decoding (Mirror-SD), an inference algorithm that breaks the latency-acceptance tradeoff.
  arXiv  Detail & Related papers  (2025-10-15T05:22:57Z)
• Slow-Fast Policy Optimization: Reposition-Before-Update for LLM Reasoning [45.51804571136028]
  Reinforcement learning (RL) has become central to enhancing reasoning in large language models (LLMs)<n>We introduce Slow-Fast Policy Optimization (SFPO), a simple yet efficient framework to address these limitations via decomposing each step into three stages.<n>SFPO consistently improves stability, reduces rollouts, and accelerates convergence of reasoning RL training.
  arXiv  Detail & Related papers  (2025-10-05T07:22:54Z)
• DiffuSpec: Unlocking Diffusion Language Models for Speculative Decoding [66.40658898418316]
  We present DiffuSpec, a training-free drop-in framework that uses a pretrained diffusion language model (DLM) to produce multi-token drafts in a single forward pass.<n>Across benchmarks, DiffuSpec yields up to 3x wall-clock speedup, establishing diffusion-based drafting as a robust alternative to autoregressive drafters for speculative decoding.
  arXiv  Detail & Related papers  (2025-09-28T07:00:15Z)
• FastLR: Non-Autoregressive Lipreading Model with Integrate-and-Fire [74.04394069262108]
  We propose FastLR, a non-autoregressive (NAR) lipreading model which generates all target tokens simultaneously. FastLR achieves the speedup up to 10.97$times$ compared with state-of-the-art lipreading model.
  arXiv  Detail & Related papers  (2020-08-06T08:28:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.