Prompt replay: speeding up grpo with on-policy reuse of high-signal prompts

• URL: http://arxiv.org/abs/2603.21177v1
• Date: Sun, 22 Mar 2026 11:39:51 GMT
• Title: Prompt replay: speeding up grpo with on-policy reuse of high-signal prompts
• Abstract summary: Reinforcement learning with verifiable rewards (RLVR) plays a crucial role in expanding the capacities of reasoning, but GRPO-style training is dominated by expensive rollouts and wastes compute on unusable prompts.<n>We propose Prompt Replay, an overhead-free online data selection method for GRPO that reuses prompts only (not trajectories) to preserve on-policy optimization.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reinforcement learning with verifiable rewards (RLVR) plays a crucial role in expanding the capacities of LLM reasoning, but GRPO-style training is dominated by expensive rollouts and wastes compute on unusable prompts. We propose Prompt Replay, an overhead-free online data selection method for GRPO that reuses prompts only (not trajectories), to preserve on-policy optimization. After each step, we insert prompts with medium difficulty into a buffer, and prioritize prompts closer to a pass rate of 0.5 (half answers correct, half wrong) to maximize the advantage, thus learning signal. Training batches are formed by mixing reused prompts with fresh samples, with cooldown steps and max reuse times controlling aggressiveness vs risk of overfitting. Across multiple model families (Llama-3.2- 3B, Qwen3-8B) and training datasets (Dolci, Polaris), evaluated using average accuracy on six standard math benchmarks, Prompt Replay reduces zero-variance prompts, increases mean absolute advantage and shows faster initial accuracy gains. Yet, it plateaus and converges with the baseline, as too aggressive configuration was used. The method is most efficient when the rollouts are the primary bottleneck and the dataset is difficult for the model. We additionally observe that Qwen2.5-Math can exhibit spurious-reward effects that invalidates ablations, raising a warning signal for using it as a sole testbed for GRPO method research.

Related papers

• Verifier-Induced Support Reshaping in On-Policy Optimization [27.782412748608255]
  We show that on-policy reinforcement learning with verifiable rewards (RLVR) can improve the current objective while making successful behaviors for later objectives too rare to sample and reinforce.<n>We call this verifier-induced support reshaping and define effective rewardable support as successful trajectories reachable within a fixed rollout budget.
  arXiv  Detail & Related papers  (2026-07-31T19:05:43Z)
• LEEPS: Latent-Guided Explore-Exploit Prompt Sampling for Efficient RLVR in Large Language Models [21.28794880416211]
  Reinforcement learning with verifiable rewards (RLVR) improves the reasoning capabilities of large language models.<n>Existing pre-rollout methods struggle to balance exploitation and exploration.<n>We introduce LEEPS, a Latent-Guided Explore--Exploit Prompt Sampler.
  arXiv  Detail & Related papers  (2026-07-30T11:50:10Z)
• REVES: REvision and VErification--Augmented Training for Test-Time Scaling [53.197756110943395]
  We propose a two-stage iterative framework that alternates between online data/prompt augmentation and policy optimization.<n>We observe gains of +6.5 points over the RL baseline and +4.0 points over standard multi-turn training.
  arXiv  Detail & Related papers  (2026-06-17T10:37:23Z)
• Replay What Matters: Off-Policy Replay for Efficient LLM Reinforcement Unlearning [19.683904518981624]
  ReRULE is an off-policy replay enhancement for reinforcement unlearning.<n>We show that ReRULE yields a tighter hard-case convergence bound than pure on-policy RULE.
  arXiv  Detail & Related papers  (2026-06-13T14:52:40Z)
• Prune as You Generate: Online Rollout Pruning for Faster and Better RLVR [78.88799271719365]
  We introduce arrol (Accelerating RLVR via online Rollout Pruning), an online rollout pruning method.<n>Arrol trains a lightweight quality head on-the-fly to predict the success probability of partial rollouts and uses it to make early pruning decisions.<n>Across GRPO and DAPO on Qwen-3 and LLaMA-3.2 models, arrol improves average accuracy by +2.30 to +2.99 while achieving up to 1.7x training speedup.
  arXiv  Detail & Related papers  (2026-03-25T22:10:36Z)
• HeaPA: Difficulty-Aware Heap Sampling and On-Policy Query Augmentation for LLM Reinforcement Learning [78.12979615107564]
  HeaPA consistently improves accuracy and reaches target performance with fewer computations.<n>Our analyses suggest these gains come from frontier-focused sampling and on-policy pool growth.
  arXiv  Detail & Related papers  (2026-01-30T01:31:17Z)
• Reinforcement Learning via Self-Distillation [37.078107691613155]
  Large language models are increasingly post-trained with reinforcement learning in verifiable domains such as code and math.<n>Current methods for reinforcement learning with verifiable rewards (RLVR) learn only from a scalar outcome reward per attempt, creating a severe credit-assignment bottleneck.<n>We formalize this setting as reinforcement learning with rich feedback and introduce Self-Distillation Policy Optimization (SDPO)<n>SDPO converts tokenized feedback into a dense learning signal without any external teacher or explicit reward model.
  arXiv  Detail & Related papers  (2026-01-28T17:45:12Z)
• Reuse your FLOPs: Scaling RL on Hard Problems by Conditioning on Very Off-Policy Prefixes [22.721425502443253]
  We introduce PrefixRL, where we condition on the prefix of successful off-policy traces and run on-policy RL to complete them.<n>PrefixRL boosts the learning signal on hard problems by modulating the difficulty of the problem through the off-policy prefix length.<n>We prove that the PrefixRL objective is not only consistent with the standard RL objective but also more efficient.
  arXiv  Detail & Related papers  (2026-01-26T18:57:00Z)
• Explore Data Left Behind in Reinforcement Learning for Reasoning Language Models [61.78513830395669]
  Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as an effective approach for improving the reasoning abilities of large language models (LLMs)<n>As models train longer and scale larger, more training prompts become residual prompts, those with zero variance rewards that provide no training signal.<n>We propose the Explore Residual Prompts in Policy Optimization framework, which encourages exploration on residual prompts and reactivates their training signals.
  arXiv  Detail & Related papers  (2025-11-06T20:40:27Z)
• Can Prompt Difficulty be Online Predicted for Accelerating RL Finetuning of Reasoning Models? [65.18157595903124]
  This work investigates iterative approximate evaluation for arbitrary prompts.<n>It introduces Model Predictive Prompt Selection (MoPPS), a Bayesian risk-predictive framework.<n>MoPPS reliably predicts prompt difficulty and accelerates training with significantly reduced rollouts.
  arXiv  Detail & Related papers  (2025-07-07T03:20:52Z)
• Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rollouts [28.17839625646103]
  We show that a substantial portion of computational overhead can be avoided by skipping uninformative prompts before rollout.<n>We propose GRESO, an online, lightweight pre-rollout filtering algorithm that predicts and skips uninformative prompts.<n>We show that GRESO achieves up to 2.4x wall-clock time speedup in rollout and up to 2.0x speedup in total training time without accuracy degradation.
  arXiv  Detail & Related papers  (2025-06-02T19:03:00Z)
• Trajectory Balance with Asynchrony: Decoupling Exploration and Learning for Fast, Scalable LLM Post-Training [71.16258800411696]
  Reinforcement learning (RL) is a critical component of large language model (LLM) post-training.<n>Existing on-policy algorithms used for post-training are inherently incompatible with the use of experience replay buffers.<n>We propose efficiently obtaining this benefit of replay buffers via Trajectory Balance with Asynchrony (TBA)
  arXiv  Detail & Related papers  (2025-03-24T17:51:39Z)
• Scalable Reinforcement Post-Training Beyond Static Human Prompts: Evolving Alignment via Asymmetric Self-Play [52.3079697845254]
  eva is the first method that allows language models to adaptively create training prompts in both offline and online RL post-training.<n>We show eva can create effective RL curricula and is robust across ablations.
  arXiv  Detail & Related papers  (2024-10-31T08:15:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.