Learning-Zone Energy: Online Data Selection for Efficient RL Post-Training

• URL: http://arxiv.org/abs/2605.17003v2
• Date: Tue, 19 May 2026 05:34:27 GMT
• Title: Learning-Zone Energy: Online Data Selection for Efficient RL Post-Training
• Abstract summary: We propose a theoretically grounded, fully online data selection framework that concentrates on the model's active learning frontier.<n>A forward pruner with replay further reduces wall-clock time cost by skipping rollout generation for persistently solved prompts.<n>Our method retains only 40% of the training data per step yet matches or surpasses full-data baselines.
• Score: 28.00739954235118
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reinforcement Learning (RL) post-training has emerged as the dominant paradigm for eliciting mathematical reasoning in Large Language Models (LLMs), yet prevailing techniques such as GRPO and DAPO distribute rollout and gradient budgets nearly uniformly across prompts, squandering compute on samples that are already mastered or remain far beyond the model's current capability. To address this fundamental inefficiency, we propose Learning-Zone Energy (LZE), a theoretically grounded, fully online data selection framework that concentrates computation on the model's active learning frontier. At its core, we define a closed-form Learning-Zone Energy Score that fuses three complementary signals, an initial-difficulty anchor, a normalized outcome-uncertainty term, and a pass-rate momentum, into a single scalar that is provably aligned with the expected magnitude of group-relative policy gradient updates. A forward pruner with replay further reduces wall-clock time cost by skipping rollout generation for persistently solved prompts while periodically checking for forgetting. Evaluated on Qwen-family models (1.5B-8B) across GSM8K, MATH and DAPO-MATH, our method retains only 40% of the training data per step yet matches or surpasses full-data baselines, with especially pronounced out-of-distribution gains on AIME25 (+45.9%) and AMC23 (+18.2%), alongside an estimated 36% reduction in training FLOPs. Our code is available at https://github.com/Stellaris167/LZE.

Related papers

• Sampling via Decision-Flow: Training-Free Extraction of Improved Latent Reasoning Paths in Large Language Models [1.8231916504534451]
  Decision-Flow Sampling (DF-Sample) is a training-free, data-free inference-time framework.<n>DF-Sample consistently outperforms baselines across three models and four benchmarks.
  arXiv  Detail & Related papers  (2026-09-11T00:48:45Z)
• Learn Where Outcomes Diverge: Efficient VLA RL via Probabilistic Chunk Masking [5.238545250784642]
  This paper presents Probabilistic Chunk Masking (PCM), a drop-in modification to GRPO that allocates gradient to a small, probabilistically selected subset of chunks per trajectory.<n>On three LIBERO benchmarks, PCM matches the final success rate of standard GRPO while achieving 2.38 times wall-clock speedup, 4.8 times faster gradient updates, and 60% lower peak activation memory.
  arXiv  Detail & Related papers  (2026-05-15T16:33:59Z)
• Mashup Learning: Faster Finetuning by Remixing Past Checkpoints [11.36221733394798]
  Mashup Learning is a simple method to leverage the outputs of prior training runs to enhance model adaptation to new tasks.<n>It consistently improves average downstream accuracy by 0.5-5 percentage points over training from scratch.<n>It also accelerates convergence, requiring 41-46% fewer training steps and up to 37% less total wall-clock time to match from-scratch accuracy.
  arXiv  Detail & Related papers  (2026-03-10T18:46:26Z)
• GradAlign: Gradient-Aligned Data Selection for LLM Reinforcement Learning [55.03441672267886]
  We propose GradAlign, a gradient-aligned data selection method for reinforcement learning.<n>We evaluate GradAlign across three data regimes: unreliable reward signals, distribution imbalance, and low-utility training corpus.
  arXiv  Detail & Related papers  (2026-02-25T01:54:50Z)
• Just-In-Time Reinforcement Learning: Continual Learning in LLM Agents Without Gradient Updates [53.3717573880076]
  We introduce Just-In-Time Reinforcement Learning (JitRL), a training-free framework that enables test-time policy optimization without any gradient updates.<n>JitRL maintains a dynamic, non-parametric memory of experiences and retrieves relevant trajectories to estimate action advantages on-the-fly.<n>Experiments on WebArena and Jericho demonstrate that JitRL establishes a new state-of-the-art among training-free methods.
  arXiv  Detail & Related papers  (2026-01-26T14:16:51Z)
• Ring-lite: Scalable Reasoning via C3PO-Stabilized Reinforcement Learning for LLMs [51.21041884010009]
  Ring-lite is a Mixture-of-Experts (MoE)-based large language model optimized via reinforcement learning (RL)<n>Our approach matches the performance of state-of-the-art (SOTA) small-scale reasoning models on challenging benchmarks.
  arXiv  Detail & Related papers  (2025-06-17T17:12:34Z)
• Efficient Continual Pre-training by Mitigating the Stability Gap [68.49269649759005]
  We study the behavior of Large Language Models (LLMs) during continual pre-training. We propose three effective strategies to enhance LLM performance within a fixed compute budget. Our strategies improve the average medical task performance of the OpenLlama-3B model from 36.2% to 40.7% with only 40% of the original training budget.
  arXiv  Detail & Related papers  (2024-06-21T02:28:37Z)
• A Meta-Learning Approach to Predicting Performance and Data Requirements [163.4412093478316]
  We propose an approach to estimate the number of samples required for a model to reach a target performance. We find that the power law, the de facto principle to estimate model performance, leads to large error when using a small dataset. We introduce a novel piecewise power law (PPL) that handles the two data differently.
  arXiv  Detail & Related papers  (2023-03-02T21:48:22Z)
• MUSBO: Model-based Uncertainty Regularized and Sample Efficient Batch Optimization for Deployment Constrained Reinforcement Learning [108.79676336281211]
  Continuous deployment of new policies for data collection and online learning is either cost ineffective or impractical. We propose a new algorithmic learning framework called Model-based Uncertainty regularized and Sample Efficient Batch Optimization. Our framework discovers novel and high quality samples for each deployment to enable efficient data collection.
  arXiv  Detail & Related papers  (2021-02-23T01:30:55Z)
• Weight Update Skipping: Reducing Training Time for Artificial Neural Networks [0.30458514384586394]
  We propose a new training methodology for ANNs that exploits the observation of improvement of accuracy shows temporal variations. During such time windows, we keep updating bias which ensures the network still trains and avoids overfitting. Such a training approach virtually achieves the same accuracy with considerably less computational cost, thus lower training time.
  arXiv  Detail & Related papers  (2020-12-05T15:12:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.