RLEP: Reinforcement Learning with Experience Replay for LLM Reasoning

• URL: http://arxiv.org/abs/2507.07451v1
• Date: Thu, 10 Jul 2025 05:58:55 GMT
• Title: RLEP: Reinforcement Learning with Experience Replay for LLM Reasoning
• Abstract summary: Reinforcement learning (RL) for large language models is an energy-intensive endeavor.<n>We present emphRLEP, a framework that first collects verified trajectories and then replays them during subsequent training.<n>At every update step, the policy is optimized on mini-batches that blend newly generated rollouts with replayed successes.
• Score: 18.62575670251997
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reinforcement learning (RL) for large language models is an energy-intensive endeavor: training can be unstable, and the policy may gradually drift away from its pretrained weights. We present \emph{RLEP}\, -- \,Reinforcement Learning with Experience rePlay\, -- \,a two-phase framework that first collects verified trajectories and then replays them during subsequent training. At every update step, the policy is optimized on mini-batches that blend newly generated rollouts with these replayed successes. By replaying high-quality examples, RLEP steers the model away from fruitless exploration, focuses learning on promising reasoning paths, and delivers both faster convergence and stronger final performance. On the Qwen2.5-Math-7B base model, RLEP reaches baseline peak accuracy with substantially fewer updates and ultimately surpasses it, improving accuracy on AIME-2024 from 38.2% to 39.9%, on AIME-2025 from 19.8% to 22.3%, and on AMC-2023 from 77.0% to 82.2%. Our code, datasets, and checkpoints are publicly available at https://github.com/Kwai-Klear/RLEP to facilitate reproducibility and further research.

Related papers

• Learning-Zone Energy: Online Data Selection for Efficient RL Post-Training [28.00739954235118]
  We propose a theoretically grounded, fully online data selection framework that concentrates on the model's active learning frontier.<n>A forward pruner with replay further reduces wall-clock time cost by skipping rollout generation for persistently solved prompts.<n>Our method retains only 40% of the training data per step yet matches or surpasses full-data baselines.
  arXiv  Detail & Related papers  (2026-05-16T14:01:12Z)
• Beyond Reasoning: Reinforcement Learning Unlocks Parametric Knowledge in LLMs [59.62441340813425]
  We study whether reinforcement learning can improve direct recall of parametric knowledge.<n>Across three model families and multiple factual QA benchmarks, RL yields 27% average relative gains.<n>Mechanistically, RL primarily redistributes probability mass over existing knowledge rather than acquiring new facts.
  arXiv  Detail & Related papers  (2026-05-08T02:40:12Z)
• Mashup Learning: Faster Finetuning by Remixing Past Checkpoints [11.36221733394798]
  Mashup Learning is a simple method to leverage the outputs of prior training runs to enhance model adaptation to new tasks.<n>It consistently improves average downstream accuracy by 0.5-5 percentage points over training from scratch.<n>It also accelerates convergence, requiring 41-46% fewer training steps and up to 37% less total wall-clock time to match from-scratch accuracy.
  arXiv  Detail & Related papers  (2026-03-10T18:46:26Z)
• QuRL: Efficient Reinforcement Learning with Quantized Rollout [23.326106976928898]
  Reinforcement learning with verifiable rewards (RLVR) has become a trending paradigm for training reasoning large language models (LLMs)<n>Due to the autoregressive decoding nature of LLMs, the rollout process becomes the efficiency bottleneck of RL training, consisting of up to 70% of the total training time.<n>We propose Quantized Reinforcement Learning (QuRL) that uses a quantized actor for accelerating the rollout.
  arXiv  Detail & Related papers  (2026-02-15T01:48:10Z)
• Just-In-Time Reinforcement Learning: Continual Learning in LLM Agents Without Gradient Updates [53.3717573880076]
  We introduce Just-In-Time Reinforcement Learning (JitRL), a training-free framework that enables test-time policy optimization without any gradient updates.<n>JitRL maintains a dynamic, non-parametric memory of experiences and retrieves relevant trajectories to estimate action advantages on-the-fly.<n>Experiments on WebArena and Jericho demonstrate that JitRL establishes a new state-of-the-art among training-free methods.
  arXiv  Detail & Related papers  (2026-01-26T14:16:51Z)
• Representation-Based Exploration for Language Models: From Test-Time to Post-Training [50.144031964319424]
  Reinforcement learning (RL) promises to expand the capabilities of language models.<n>It is unclear if current RL techniques promote the discovery of novel behaviors, or simply sharpen those already present in the base model.<n>We investigate the value of deliberate exploration -- explicitly incentivizing the model to discover novel and diverse behaviors.
  arXiv  Detail & Related papers  (2025-10-13T17:49:05Z)
• Squeeze the Soaked Sponge: Efficient Off-policy Reinforcement Finetuning for Large Language Model [56.92219181993453]
  We propose Reincarnating Mix-policy Proximal Policy Gradient (ReMix) to enable on-policyRFT methods like PPO and GRPO to leverage off-policy data.<n>ReMix consists of three major components: (1) Mix-policy proximal policy gradient with an increased Update-To-Data (UTD) ratio for efficient training; (2) KL-Convex policy constraint to balance the trade-off between stability and flexibility; (3) Policy reincarnation to achieve a seamless transition from efficient early-stage learning to steady improvement.
  arXiv  Detail & Related papers  (2025-07-09T14:29:45Z)
• Confidence Is All You Need: Few-Shot RL Fine-Tuning of Language Models [43.98994504606355]
  We propose Reinforcement Learning via Self-Confidence (RLSC) for large language models (LLMs)<n>RLSC uses the model's own confidence as reward signals-eliminating the need for labels, preference models, or reward engineering.
  arXiv  Detail & Related papers  (2025-06-05T19:55:15Z)
• AceReason-Nemotron: Advancing Math and Code Reasoning through Reinforcement Learning [50.02117478165099]
  We show that large-scale reinforcement learning can significantly enhance the reasoning capabilities of strong, small- and mid-sized models.<n>We propose a simple yet effective approach: first training on math-only prompts, then on code-only prompts.
  arXiv  Detail & Related papers  (2025-05-22T08:50:47Z)
• Think Twice: Enhancing LLM Reasoning by Scaling Multi-round Test-time Thinking [16.441081996257576]
  We propose a simple yet effective test-time scaling approach Multi-round Thinking.<n>This method iteratively refines model reasoning by leveraging previous answers as prompts for subsequent rounds.<n>Experiments across multiple models, including QwQ-32B and DeepSeek-R1, consistently show performance improvements.
  arXiv  Detail & Related papers  (2025-03-25T17:19:38Z)
• An Empirical Study on Eliciting and Improving R1-like Reasoning Models [90.52239241349504]
  scaling RL training has become a central technique for implementing such reasoning models.<n>We demonstrate that our RL training approach consistently improves the Qwen2.5-32B base models.<n>We also explore the use of tool manipulation, finding that it significantly boosts the reasoning performance of large reasoning models.
  arXiv  Detail & Related papers  (2025-03-06T15:34:27Z)
• LIMR: Less is More for RL Scaling [25.477841726836836]
  We introduce Learning Impact Measurement (LIM), an automated method to evaluate and prioritize training samples.<n>Our method achieves comparable or even superior performance using only 1,389 samples versus the full 8,523 samples dataset.<n>For reproducible research and future innovation, we are open-sourcing LIMR, including implementation of LIM, training and evaluation code, curated datasets, and trained models.
  arXiv  Detail & Related papers  (2025-02-17T15:13:29Z)
• Kimi k1.5: Scaling Reinforcement Learning with LLMs [84.95584393629998]
  We report on the training practice of Kimi k1.5, our latest multi-modal language model trained with reinforcement learning.<n>Long context scaling and improved policy optimization methods are key ingredients of our approach.<n>Our system achieves state-of-the-art reasoning performance across multiple benchmarks and modalities.
  arXiv  Detail & Related papers  (2025-01-22T02:48:14Z)
• Process Supervision-Guided Policy Optimization for Code Generation [15.943210767010045]
  Reinforcement learning (RL) with unit test feedback has enhanced large language models' (LLMs) code generation, but relies on sparse rewards provided only after complete code evaluation.<n>We propose a Process Reward Model (PRM) that delivers dense, line-level feedback on code correctness during generation, mimicking human code refinement.
  arXiv  Detail & Related papers  (2024-10-23T07:22:33Z)
• VinePPO: Refining Credit Assignment in RL Training of LLMs [66.80143024475635]
  We propose VinePPO, a straightforward approach that leverages the flexibility of language environments to compute unbiased Monte Carlo-based estimates.<n>Our method consistently outperforms PPO and other baselines across MATH and GSM8K datasets in less wall-clock time.
  arXiv  Detail & Related papers  (2024-10-02T15:49:30Z)
• Learning to Prune Deep Neural Networks via Reinforcement Learning [64.85939668308966]
  PuRL is a deep reinforcement learning based algorithm for pruning neural networks. It achieves sparsity and accuracy comparable to current state-of-the-art methods.
  arXiv  Detail & Related papers  (2020-07-09T13:06:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.