LatentRevise: Learning from Zero-Hit Reasoning

• URL: http://arxiv.org/abs/2606.29938v1
• Date: Mon, 29 Jun 2026 08:14:47 GMT
• Title: LatentRevise: Learning from Zero-Hit Reasoning
• Abstract summary: Reinforcement learning with verifiable rewards is bottlenecked by hard prompts on which correct trajectories have low probability.<n>We frame such zero-hit prompts as RLVR's sampling frontier, where new reasoning behavior is most valuable yet least likely to be sampled.<n>We introduce LatentRevise, a first-order latent revision method that recovers training signal for this zero-hit regime.
• Score: 46.1536368072511
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reinforcement learning with verifiable rewards (RLVR) is bottlenecked by hard prompts on which correct trajectories have low probability, so sampling misses them within a practical budget and leaves the policy update with little useful signal. We frame such zero-hit prompts as RLVR's sampling frontier, where new reasoning behavior is most valuable yet least likely to be sampled. Importantly, failed rollouts can be informative: they expose where the model's reasoning went wrong. We introduce LatentRevise, a first-order latent revision method that recovers training signal for this zero-hit regime. Given a failed rollout and the gold answer as an anchor, LatentRevise optimizes the input embeddings of its reasoning prefix under two complementary gradients, moving the prefix away from the failed continuation and toward the gold answer. The optimization is constrained to the convex hull of the model's vocabulary embeddings, so each update moves the latent toward a real token embedding rather than an arbitrary feature direction. We find that continuations from the revised prefix lengthen, exhibit self-reflection, and reach correct answers missed by the original rollouts. Used as training data, these trajectories improve SFT and RLVR on math benchmarks over standard baselines.

Related papers

• Verifier-Induced Support Reshaping in On-Policy Optimization [27.782412748608255]
  We show that on-policy reinforcement learning with verifiable rewards (RLVR) can improve the current objective while making successful behaviors for later objectives too rare to sample and reinforce.<n>We call this verifier-induced support reshaping and define effective rewardable support as successful trajectories reachable within a fixed rollout budget.
  arXiv  Detail & Related papers  (2026-07-31T19:05:43Z)
• REVES: REvision and VErification--Augmented Training for Test-Time Scaling [53.197756110943395]
  We propose a two-stage iterative framework that alternates between online data/prompt augmentation and policy optimization.<n>We observe gains of +6.5 points over the RL baseline and +4.0 points over standard multi-turn training.
  arXiv  Detail & Related papers  (2026-06-17T10:37:23Z)
• Understanding Diversity Collapse in RLVR via the Lens of Overtraining [78.37408098404312]
  Reinforcement learning with verifiable rewards (RLVR) has become a key approach for enhancing the reasoning abilities of large language models.<n>We formalize this diversity collapse through the lens of emphovertraining<n>We propose emphBayesian Boundary Gating (BBG) which redirects optimization away from overtraining by estimating each problem's marginal contribution to the reasoning boundary.
  arXiv  Detail & Related papers  (2026-06-13T20:13:37Z)
• Learning to Solve, Forgetting to Retain: Correct-Set Turnover in RLVR [16.681943272535104]
  Reinforcement learning with verifiable rewards (RLVR) improves the ability of large language model, yet headline accuracy gains often conceal a hidden cost.<n>We propose textbfmethod, a retention-aware review mechanism that tracks mastered prompts and periodically reintroduces them to textbfremind the model of previous solutions.
  arXiv  Detail & Related papers  (2026-06-02T03:17:34Z)
• Adaptive Negative Reinforcement for LLM Reasoning:Dynamically Balancing Correction and Diversity in RLVR [6.291433683451589]
  Reinforcement learning with verifiable rewards (RLVR) has become a highly effective method for improving the reasoning abilities of Large Language Models (LLMs)<n>Recent research shows that Negative Sample Reinforcement (NSR) can match or even exceed the performance of more complex frameworks like PPO and GRPO.<n>We propose two extensions to the NSR framework: Adaptive Negative Sample Reinforcement and Confidence-Weighted Negative Reinforcement.
  arXiv  Detail & Related papers  (2026-05-08T02:13:33Z)
• How Far Can Unsupervised RLVR Scale LLM Training? [57.44753418846446]
  Unsupervised reinforcement learning with verifiable rewards (URLVR) offers a pathway to scale LLM training beyond the supervision bottleneck.<n>Recent works leverage model intrinsic signals, showing promising early gains, yet their potential and limitations remain unclear.<n>We classify URLVR methods into intrinsic versus external based on reward sources, then establish a unified theoretical framework.
  arXiv  Detail & Related papers  (2026-03-09T17:38:11Z)
• Reinforcement Learning via Self-Distillation [37.078107691613155]
  Large language models are increasingly post-trained with reinforcement learning in verifiable domains such as code and math.<n>Current methods for reinforcement learning with verifiable rewards (RLVR) learn only from a scalar outcome reward per attempt, creating a severe credit-assignment bottleneck.<n>We formalize this setting as reinforcement learning with rich feedback and introduce Self-Distillation Policy Optimization (SDPO)<n>SDPO converts tokenized feedback into a dense learning signal without any external teacher or explicit reward model.
  arXiv  Detail & Related papers  (2026-01-28T17:45:12Z)
• From Verifiable Dot to Reward Chain: Harnessing Verifiable Reference-based Rewards for Reinforcement Learning of Open-ended Generation [52.62655622099456]
  We propose reinforcement learning with verifiable reference-based rewards (RLVRR)<n>Instead of checking the final answer, RLVRR extracts an ordered linguistic signal from high-quality references (i.e., reward chain)<n>In this way, RLVRR decomposes rewards into two dimensions: content, which preserves deterministic core concepts, and style, which evaluates adherence to stylistic properties.
  arXiv  Detail & Related papers  (2026-01-26T14:39:58Z)
• Reinforcement Learning with Verifiable yet Noisy Rewards under Imperfect Verifiers [90.50039419576807]
  Reinforcement Learning with Verifiable Rewards (RLVR) trains policies against automated verifiers to avoid costly human labeling.<n>To reduce vulnerability to verifier hacking, many RLVR systems collapse rewards to binary $0,1$ during training.<n>This choice carries a cost: it introduces textitfalse negatives (rejecting correct answers, FNs) and textitfalse positives (accepting incorrect ones, FPs)
  arXiv  Detail & Related papers  (2025-10-01T13:56:44Z)
• The Surprising Effectiveness of Negative Reinforcement in LLM Reasoning [37.13807960501503]
  Reinforcement learning with verifiable rewards (RLVR) is a promising approach for training language models (LMs)<n>We decompose the learning signal into reinforcing correct responses and penalizing incorrect ones, referred to as Positive and Negative Sample Reinforcement (PSR and NSR)<n>We show that NSR works by suppressing incorrect generations and redistributing probability mass toward other plausible candidates, guided by the model's prior beliefs.
  arXiv  Detail & Related papers  (2025-06-02T06:10:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.