Train at Moving Edge: Online-Verified Prompt Selection for Efficient RL Training of Large Reasoning Model

• URL: http://arxiv.org/abs/2603.25184v1
• Date: Thu, 26 Mar 2026 08:52:35 GMT
• Title: Train at Moving Edge: Online-Verified Prompt Selection for Efficient RL Training of Large Reasoning Model
• Abstract summary: HIVE is a dual-stage framework for data-efficientReinforcement learning.<n>We show that HIVE yields significant rollout efficiency without compromising performance.
• Score: 12.851049183155482
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reinforcement learning (RL) has become essential for post-training large language models (LLMs) in reasoning tasks. While scaling rollouts can stabilize training and enhance performance, the computational overhead is a critical issue. In algorithms like GRPO, multiple rollouts per prompt incur prohibitive costs, as a large portion of prompts provide negligible gradients and are thus of low utility. To address this problem, we investigate how to select high-utility prompts before the rollout phase. Our experimental analysis reveals that sample utility is non-uniform and evolving: the strongest learning signals concentrate at the ``learning edge", the intersection of intermediate difficulty and high uncertainty, which shifts as training proceeds. Motivated by this, we propose HIVE (History-Informed and online-VErified prompt selection), a dual-stage framework for data-efficient RL. HIVE utilizes historical reward trajectories for coarse selection and employs prompt entropy as a real-time proxy to prune instances with stale utility. By evaluating HIVE across multiple math reasoning benchmarks and models, we show that HIVE yields significant rollout efficiency without compromising performance.

Related papers

• Unified Data Selection for LLM Reasoning [66.0118347918729]
  Training Large Language Models (LLMs) for complex, long-CoT reasoning is often bottlenecked by the need for massive high-quality reasoning data.<n>We propose High-Entropy Sum (HES), a training-free metric that quantifies reasoning quality by summing only the entropy of the top (e.g., 0.5%) highest-entropy tokens in each reasoning sample.
  arXiv  Detail & Related papers  (2026-05-21T12:21:41Z)
• Dynamics-Predictive Sampling for Active RL Finetuning of Large Reasoning Models [49.04912820721943]
  Reinforcement learning (RL) finetuning has become a key technique for enhancing the reasoning abilities of large language models (LLMs)<n>Recent advances underscore the importance of online prompt selection methods, which typically concentrate training on partially solved or moderately challenging examples.<n>This work proposes Dynamics-Predictive Sampling (DPS), which online predicts and selects informative prompts by inferring their learning dynamics prior to costly rollouts.
  arXiv  Detail & Related papers  (2026-03-11T15:31:14Z)
• Tailored Primitive Initialization is the Secret Key to Reinforcement Learning [61.29280885291581]
  Reinforcement learning (RL) has emerged as a powerful paradigm for enhancing the reasoning capabilities of large language models (LLMs)<n>We argue that initializing LLMs with diverse, high-quality reasoning primitives is essential for achieving stable and sample-efficient RL training.<n>We propose Tailor, a finetuning pipeline that automatically discovers and curates novel reasoning primitives.
  arXiv  Detail & Related papers  (2025-11-16T03:12:40Z)
• HINT: Helping Ineffective Rollouts Navigate Towards Effectiveness [49.72591739116668]
  Reinforcement Learning (RL) has become a key driver for enhancing the long chain-of-thought (CoT) reasoning capabilities of Large Language Models (LLMs)<n>However, prevalent methods like GRPO often fail when task difficulty exceeds the model's capacity, leading to reward sparsity and inefficient training.<n>We propose HINT: Helping Ineffective rollouts Navigate Towards effectiveness, an adaptive hinting framework.
  arXiv  Detail & Related papers  (2025-10-10T13:42:03Z)
• CurES: From Gradient Analysis to Efficient Curriculum Learning for Reasoning LLMs [53.749193998004166]
  Curriculum learning plays a crucial role in enhancing the training efficiency of large language models.<n>We propose CurES, an efficient training method that accelerates convergence and employs Bayesian posterior estimation to minimize computational overhead.
  arXiv  Detail & Related papers  (2025-10-01T15:41:27Z)
• Towards High Data Efficiency in Reinforcement Learning with Verifiable Reward [54.708851958671794]
  We propose a Data-Efficient Policy Optimization pipeline that combines optimized strategies for both offline and online data selection.<n>In offline phase, we curate a high-quality subset of training samples based on diversity, influence, and appropriate difficulty.<n>During online RLVR training, we introduce a sample-level explorability metric to dynamically filter samples with low exploration potential.
  arXiv  Detail & Related papers  (2025-09-01T10:04:20Z)
• Can Prompt Difficulty be Online Predicted for Accelerating RL Finetuning of Reasoning Models? [65.18157595903124]
  This work investigates iterative approximate evaluation for arbitrary prompts.<n>It introduces Model Predictive Prompt Selection (MoPPS), a Bayesian risk-predictive framework.<n>MoPPS reliably predicts prompt difficulty and accelerates training with significantly reduced rollouts.
  arXiv  Detail & Related papers  (2025-07-07T03:20:52Z)
• SPEED-RL: Faster Training of Reasoning Models via Online Curriculum Learning [30.90778400005588]
  Training large language models with reinforcement learning (RL) against verifiable rewards significantly enhances their reasoning abilities.<n>We introduce Selective Prompting with Efficient Estimation of Difficulty (SPEED), an adaptive online RL curriculum that selectively chooses training examples of intermediate difficulty to maximize learning efficiency.<n> Empirically, our efficient implementation leads to 2x to 6x faster training without degrading accuracy, requires no manual tuning, and integrates seamlessly into standard RL algorithms.
  arXiv  Detail & Related papers  (2025-06-10T17:42:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.