LEEPS: Latent-Guided Explore-Exploit Prompt Sampling for Efficient RLVR in Large Language Models

• URL: http://arxiv.org/abs/2607.28077v1
• Date: Thu, 30 Jul 2026 11:50:10 GMT
• Title: LEEPS: Latent-Guided Explore-Exploit Prompt Sampling for Efficient RLVR in Large Language Models
• Abstract summary: Reinforcement learning with verifiable rewards (RLVR) improves the reasoning capabilities of large language models.<n>Existing pre-rollout methods struggle to balance exploitation and exploration.<n>We introduce LEEPS, a Latent-Guided Explore--Exploit Prompt Sampler.
• Score: 21.28794880416211
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reinforcement learning with verifiable rewards (RLVR) improves the reasoning capabilities of large language models, but prompt groups with identical rollout rewards consume generation budget without effective learning signals. Pre-rollout prompt selection can reduce this waste by screening prompts before rollout generation. However, existing pre-rollout methods struggle to balance exploitation and exploration: repeatedly exploiting historically informative prompts can narrow training coverage, whereas broader exploration can lower the fraction of informative prompts. To address these limitations, we introduce LEEPS, a Latent-Guided Explore--Exploit Prompt Sampler that adaptively balances the reuse of previously observed informative prompts with continued exploration of uncertain ones. LEEPS partitions candidates into exploit and explore portfolios and adaptively allocates rollout budget according to their recent non-trivial ratios. It further uses representation-space neighbors and historical rollout outcomes to prioritize uncertain prompts likely to yield non-zero reward variance, thereby making exploration more targeted without additional rollouts. Across six mathematical reasoning benchmarks, LEEPS achieves the highest average score at both model scales, with relative gains of 2.6\% and 3.7\% over the strongest baseline for Qwen2.5-Math-1.5B and 7B, respectively, and generally improves faster during the training process. It also achieves the highest average score across the three evaluated OOD general-reasoning benchmarks at both model scales and adds only about 2 seconds of online sampling overhead per training step. Code is available at https://github.com/ShuangLiangX/LEEPS.

Related papers

• ExTra: Exploratory Trajectory Optimization for Language Model Reinforcement Learning [75.26064451955199]
  ExTra is a framework that extracts exploration signals from the model's own rollouts.<n>It improves Qwen3-1.7B over GRPO by about +5 points on pass@1 and +7 points on pass@16.
  arXiv  Detail & Related papers  (2026-06-23T15:51:39Z)
• Prune as You Generate: Online Rollout Pruning for Faster and Better RLVR [78.88799271719365]
  We introduce arrol (Accelerating RLVR via online Rollout Pruning), an online rollout pruning method.<n>Arrol trains a lightweight quality head on-the-fly to predict the success probability of partial rollouts and uses it to make early pruning decisions.<n>Across GRPO and DAPO on Qwen-3 and LLaMA-3.2 models, arrol improves average accuracy by +2.30 to +2.99 while achieving up to 1.7x training speedup.
  arXiv  Detail & Related papers  (2026-03-25T22:10:36Z)
• Prompt replay: speeding up grpo with on-policy reuse of high-signal prompts [0.0]
  Reinforcement learning with verifiable rewards (RLVR) plays a crucial role in expanding the capacities of reasoning, but GRPO-style training is dominated by expensive rollouts and wastes compute on unusable prompts.<n>We propose Prompt Replay, an overhead-free online data selection method for GRPO that reuses prompts only (not trajectories) to preserve on-policy optimization.
  arXiv  Detail & Related papers  (2026-03-22T11:39:51Z)
• Generative Reasoning Re-ranker [24.386586034456673]
  Generative Reasoning Reranker (GR2) is an end-to-end framework with a three-stage training pipeline tailored for reranking.<n> GR2 generates high-quality reasoning traces through carefully designed prompting and rejection sampling.<n>Experiments on two real-world datasets demonstrate GR2's effectiveness.
  arXiv  Detail & Related papers  (2026-02-08T02:12:24Z)
• Explore Data Left Behind in Reinforcement Learning for Reasoning Language Models [61.78513830395669]
  Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as an effective approach for improving the reasoning abilities of large language models (LLMs)<n>As models train longer and scale larger, more training prompts become residual prompts, those with zero variance rewards that provide no training signal.<n>We propose the Explore Residual Prompts in Policy Optimization framework, which encourages exploration on residual prompts and reactivates their training signals.
  arXiv  Detail & Related papers  (2025-11-06T20:40:27Z)
• Efficient Reinforcement Learning for Large Language Models with Intrinsic Exploration [33.02780998281276]
  Reinforcement learning with verifiable rewards (RLVR) has improved the reasoning ability of large language models.<n>This study investigates how simply leveraging intrinsic data properties, almost free benefit during training, can improve data efficiency for RLVR.
  arXiv  Detail & Related papers  (2025-11-02T04:16:47Z)
• Reinforcement Learning on Pre-Training Data [55.570379963147424]
  We introduce Reinforcement Learning on Pre-Training data (R), a new training-time scaling paradigm for optimizing large language models (LLMs)<n>R enables the policy to autonomously explore meaningful trajectories to learn from pre-training data and improve its capability through reinforcement learning (RL)<n>Extensive experiments on both general-domain and mathematical reasoning benchmarks across multiple models validate the effectiveness of R.
  arXiv  Detail & Related papers  (2025-09-23T17:10:40Z)
• Breaking the Exploration Bottleneck: Rubric-Scaffolded Reinforcement Learning for General LLM Reasoning [46.610146536866445]
  Large Language Models (LLMs) have underscored the potential of Reinforcement Learning (RL) to facilitate the emergence of reasoning capabilities.<n>We proposeScaffolded Reinforcement Learning (RuscaRL) to break the exploration bottleneck for general reasoning.<n>We show that RuscaRL significantly boosts Qwen2.5-7B-Instruct from 23.6 to 50.3 on HealthBench-500, surpassing GPT-4.1.
  arXiv  Detail & Related papers  (2025-08-23T08:47:31Z)
• Can Prompt Difficulty be Online Predicted for Accelerating RL Finetuning of Reasoning Models? [65.18157595903124]
  This work investigates iterative approximate evaluation for arbitrary prompts.<n>It introduces Model Predictive Prompt Selection (MoPPS), a Bayesian risk-predictive framework.<n>MoPPS reliably predicts prompt difficulty and accelerates training with significantly reduced rollouts.
  arXiv  Detail & Related papers  (2025-07-07T03:20:52Z)
• Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rollouts [28.17839625646103]
  We show that a substantial portion of computational overhead can be avoided by skipping uninformative prompts before rollout.<n>We propose GRESO, an online, lightweight pre-rollout filtering algorithm that predicts and skips uninformative prompts.<n>We show that GRESO achieves up to 2.4x wall-clock time speedup in rollout and up to 2.0x speedup in total training time without accuracy degradation.
  arXiv  Detail & Related papers  (2025-06-02T19:03:00Z)
• NoisyRollout: Reinforcing Visual Reasoning with Data Augmentation [66.36912000442608]
  NoisyRollout is a simple yet effective data augmentation method.<n>It mixes training trajectories from both clean and moderately distorted images.<n>It achieves state-of-the-art performance among open-source RL-tuned models.
  arXiv  Detail & Related papers  (2025-04-17T16:10:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.