Single-Rollout Hidden-State Dynamics for Training-Free RLVR Data Selection

• URL: http://arxiv.org/abs/2605.28631v1
• Date: Wed, 27 May 2026 15:38:09 GMT
• Title: Single-Rollout Hidden-State Dynamics for Training-Free RLVR Data Selection
• Abstract summary: We study RLVR data selection in a setting where selection must be performed before any RL training.<n>We propose SHIFT, a one-shot, training-free selector based solely on inference-time hiddenstate dynamics.
• Score: 31.03997249039617
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reinforcement learning with verifiable rewards (RLVR) can yield large reasoning gains from very few training instances, yet its strong sensitivity to which instances are used makes data selection a central bottleneck. Most existing selection pipelines rely on training-time optimization signals and/or require access to verifiable rewards or ground-truth answers over large candidate pools, which is costly and often infeasible in specialized domains. We study RLVR data selection in a setting where selection must be performed before any RL training and without labels or reward evaluation on the full pool. We propose SHIFT, a one-shot, training-free selector based solely on inference-time hidden-state dynamics. For each candidate instance, SHIFT runs a single deterministic reasoning rollout and computes a reasoning-induced representation shift (RIRS) as the start-to-end hidden-state delta. SHIFT uses the RIRS magnitude as a lightweight proxy for instance utility and enforces coverage via a quality-weighted farthest-first CoreSet procedure in an RIRS-augmented feature space, producing compact subsets that scale to large unlabeled pools. Across mathematical reasoning and medical QA benchmarks under ultra-low budgets, SHIFT consistently outperforms training-free diversity and difficulty/uncertainty baselines, improving both in-domain accuracy and transfer to harder evaluation settings. Ablations show that RIRS-based coverage and quality-weighting contribute complementary gains, and analyses indicate that RIRS is not explained by simple input/output length statistics. Code is available at github.com/JianghaoWu/SHIFT.

Related papers

• Is Next-Chunk Reasoning RL Really Better than SFT? Revisiting Training Strategies under no-CoT Data [48.92748013043833]
  We show that Mixed SFT achieves a clearly higher post-RLVR performance ceiling than next-chunk reasoning RL.<n>We also show that higher pre-RLVR accuracy does not necessarily translate into higher post-RLVR accuracy.
  arXiv  Detail & Related papers  (2026-08-24T13:47:45Z)
• Stop Thinking, Start Looking: Efficient Post-Training for Multimodal Document Question Answering via Reasoning-Free Alignment [2.8730465903425877]
  We introduce Perception-RFT, a training framework that applies Group Relative Policy Optimization to multimodal document QA.<n>We find that reasoning-enabled models suppress their reasoning traces during training, converging to direct perception-based policies at the 4B parameter scale.<n>We show that an early SFT$rightarrow$RL transition achieves comparable precision with 65% less training data.
  arXiv  Detail & Related papers  (2026-07-16T07:43:39Z)
• REAR: Test-time Preference Realignment through Reward Decomposition [68.09214603569744]
  We introduce a novel framework that models the task as a realignment problem, since the base model often fails to align with the stated preference.<n>We derive a REAlignment Reward (REAR) that selectively rescales the proportions of these two reward terms.<n>We show that REAR can be formulated as a linear combination of token-level policy log-probabilities, making it computationally efficient and easy to integrate with various TTS algorithms.
  arXiv  Detail & Related papers  (2026-06-29T14:17:53Z)
• Boosting Reinforcement Learning with Verifiable Rewards via Randomly Selected Few-Shot Guidance [117.2119290254454]
  We propose FEST, a demonstration-guided RLVR algorithm.<n>It attains compelling results with only 128 demonstrations randomly selected from an SFT dataset.<n>We find that three components are vital for the success: supervised signal, on-policy signal, and decaying weights on the few-shot SFT dataset.
  arXiv  Detail & Related papers  (2026-05-14T16:12:30Z)
• Ada-RS: Adaptive Rejection Sampling for Selective Thinking [8.760287445955045]
  Large language models (LLMs) are increasingly being deployed in cost and latency-sensitive settings.<n>We study selective thinking for tool-using LLMs and introduce Adaptive Rejection Sampling (Ada-RS)<n>Ada-RS scores multiple sampled completions with an adaptive length-penalized reward then applies rejection sampling to retain only high-reward candidates.
  arXiv  Detail & Related papers  (2026-02-23T05:20:15Z)
• Learning in Context, Guided by Choice: A Reward-Free Paradigm for Reinforcement Learning with Transformers [55.33468902405567]
  We propose a new learning paradigm, In-Context Preference-based Reinforcement Learning (ICPRL), in which both pretraining and deployment rely solely on preference feedback.<n>ICPRL enables strong in-context generalization to unseen tasks, achieving performance comparable to ICRL methods trained with full reward supervision.
  arXiv  Detail & Related papers  (2026-02-09T03:42:16Z)
• Learn More with Less: Uncertainty Consistency Guided Query Selection for RLVR [18.494852448006462]
  Existing RLVR algorithms require large query budgets, making annotation costly.<n>We investigate whether fewer but more informative queries can yield similar or superior performance, introducing active learning (AL) into RLVR.<n>Experiments show our method consistently outperforms random and classic AL baselines, achieving full-dataset performance while training on only 30% of the data.
  arXiv  Detail & Related papers  (2026-01-30T05:41:55Z)
• Utility-Diversity Aware Online Batch Selection for LLM Supervised Fine-tuning [49.04912820721943]
  Supervised fine-tuning (SFT) is computationally expensive and sometimes suffers from overfitting or bias amplification.<n>This work studies the online batch selection family that dynamically scores and filters samples during the training process.<n>We develop textbfUDS (Utility-Diversity Sampling), a framework for efficient online batch selection in SFT.
  arXiv  Detail & Related papers  (2025-10-19T15:32:01Z)
• Soft Random Sampling: A Theoretical and Empirical Analysis [59.719035355483875]
  Soft random sampling (SRS) is a simple yet effective approach for efficient deep neural networks when dealing with massive data. It selects a uniformly speed at random with replacement from each data set in each epoch. It is shown to be a powerful and competitive strategy with significant and competitive performance on real-world industrial scale.
  arXiv  Detail & Related papers  (2023-11-21T17:03:21Z)
• Single-Trajectory Distributionally Robust Reinforcement Learning [21.955807398493334]
  We propose Distributionally Robust RL (DRRL) to enhance performance across a range of environments. Existing DRRL algorithms are either model-based or fail to learn from a single sample trajectory. We design a first fully model-free DRRL algorithm, called distributionally robust Q-learning with single trajectory (DRQ)
  arXiv  Detail & Related papers  (2023-01-27T14:08:09Z)
• Explaining Cross-Domain Recognition with Interpretable Deep Classifier [100.63114424262234]
  Interpretable Deep (IDC) learns the nearest source samples of a target sample as evidence upon which the classifier makes the decision. Our IDC leads to a more explainable model with almost no accuracy degradation and effectively calibrates classification for optimum reject options.
  arXiv  Detail & Related papers  (2022-11-15T15:58:56Z)
• B-Pref: Benchmarking Preference-Based Reinforcement Learning [84.41494283081326]
  We introduce B-Pref, a benchmark specially designed for preference-based RL. A key challenge with such a benchmark is providing the ability to evaluate candidate algorithms quickly. B-Pref alleviates this by simulating teachers with a wide array of irrationalities.
  arXiv  Detail & Related papers  (2021-11-04T17:32:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.