Delay, Plateau, or Collapse: Evaluating the Impact of Systematic Verification Error on RLVR

• URL: http://arxiv.org/abs/2605.02909v1
• Date: Mon, 06 Apr 2026 15:02:52 GMT
• Title: Delay, Plateau, or Collapse: Evaluating the Impact of Systematic Verification Error on RLVR
• Abstract summary: We study the impact of systematic verification errors onReinforcement Learning with Verifiable Rewards (RLVR)<n>Our results show that, in contrast to prior conclusions, realistic verification errors can critically shape RLVR outcomes.
• Score: 23.784808278172164
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) has become a powerful approach for improving the reasoning capabilities of large language models (LLMs). While RLVR is designed for tasks with verifiable ground-truth answers, real-world verifiers (e.g., static code checkers) can introduce errors into the reward signal. Prior analyses have largely treated such errors as random and independent across samples, concluding that errors merely slow training with limited effect on final performance. However, practical verifiers tend to exhibit systematic errors. This introduces a risk of models learning unwanted consistent behavior from a structurally incorrect reward signal. In this work, we study the impact of such systematic verification errors on RLVR. Through controlled experiments on arithmetic tasks, we show that systematic false negatives lead to similar effects as random noise. On the other hand, systematic false positives can cause a wide range of behaviors from sub-optimal plateaus to performance collapse. Crucially, these outcomes are not determined by the overall error rate but by the specific pattern of introduced errors, making pre-hoc mitigation difficult. Our results show that, in contrast to prior conclusions, realistic verification errors can critically shape RLVR outcomes and that verifier quality has to be understood beyond its sample-level error rate.

Related papers

• Bounding the Causal Impact of ML-assisted Decision-Making via Counterfactual Correctness [2.9267249513932008]
  We present a partial-identification approach to constructing bounds on the causal effect of a new model.<n>The core innovation in our approach is to leverage assumptions relating fine-grained predictive accuracy to downstream outcomes.
  arXiv  Detail & Related papers  (2026-07-23T20:45:10Z)
• Beyond Monotonic Progress: Retry-Supervised Value Learning for Robot Imitation [38.88183703128359]
  We propose ReTVL (ReTry-Supervised Value Learning), a framework for learning mistake-sensitive value functions from mixed-quality robot demonstrations.<n>ReTVL captures the local degradation-and-recovery structure around mistakes by combining global progress calibration with local pairwise preference learning.<n> Experiments on real-robot manipulation tasks show that ReTVL produces more fine-grained value estimates than progress-based baselines.
  arXiv  Detail & Related papers  (2026-06-23T14:27:10Z)
• Probing Embodied LLMs: When Higher Observation Fidelity Hurts Problem Solving [7.648784748888186]
  Large Language Models are proposed as cognitive components for robotic systems.<n>We study embodied LLM agents behaviorally by varying the information available to the agent.<n>We find that moderate noise improves performance, peaking at a 40% flip probability with a 2.85-fold success rate increase over the noise-free baseline.
  arXiv  Detail & Related papers  (2026-05-19T16:27:00Z)
• Leveraging Error Diversity in Group Rollouts for Reinforcement Learning [61.10980882949414]
  Reinforcement Learning from Verifiable Rewards (RLVR) typically samples multiple responses per prompt and assigns binary rewards based on individual correctness.<n>We propose Error Diversity Advantage Shaping (EDAS) to modulate the advantage signal for incorrect rollouts based on intra-group error diversity.<n>EDAS operates as a simple post-hoc adjustment that can be seamlessly integrated into any RLVR algorithm.
  arXiv  Detail & Related papers  (2026-05-17T08:52:31Z)
• Learning from Failures: Correction-Oriented Policy Optimization with Verifiable Rewards [73.44333771806282]
  Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as an effective paradigm for improving the reasoning capabilities of large language models.<n>We propose Correction-Oriented Policy Optimization (CIPO), a simple and effective extension to RLVR that converts on-policy failed trajectories into correction-oriented supervision.<n>CIPO improves learning effectiveness while explicitly enhancing the model's ability to correct its own errors.
  arXiv  Detail & Related papers  (2026-05-14T08:22:21Z)
• An Imperfect Verifier is Good Enough: Learning with Noisy Rewards [4.493881508510229]
  Reinforcement Learning with Verifiable Rewards (RLVR) has become a prominent method for post-training Large Language Models (LLMs)<n>The extent to which RLVR is robust to such noise and the verifier accuracy required for effective training remain unresolved questions.<n>We investigate these questions in the domains of code generation and scientific reasoning by introducing noise into RL training.
  arXiv  Detail & Related papers  (2026-04-09T00:15:01Z)
• Correct Answers from Sound Reasoning: Verifiable Process Supervision for Language Models [94.68358825189738]
  We propose a post-training framework for verifiable domains that jointly optimize prediction accuracy and reasoning quality.<n>We evaluate VPS on chess, a controlled testbed where reasoning steps can be deterministically verified against engine signals.<n>VPS preserves accuracy while significantly improving reasoning quality, reducing win-rate error by up to 30% and restoring consistency to near saturation.
  arXiv  Detail & Related papers  (2026-04-03T15:19:46Z)
• Decoding the Critique Mechanism in Large Reasoning Models [50.821607345799386]
  Large Reasoning Models (LRMs) exhibit backtracking and self-verification mechanisms that enable them to revise intermediate steps and reach correct solutions.<n>We investigate how current LRMs recover from errors by inserting arithmetic mistakes in their intermediate reasoning steps.<n>We find that despite the error propagating through the chain-of-thought, the model still reaches the correct final answer.
  arXiv  Detail & Related papers  (2026-03-17T10:03:30Z)
• CARE What Fails: Contrastive Anchored-REflection for Verifiable Multimodal [84.71254539482369]
  Group-relative reinforcement learning with verifiable rewards (RLVR) often wastes the most informative data it already has the failures.<n>We present CARE, a failure-centric post-training framework for multimodal reasoning that turns errors into supervision.<n> CARE improves accuracy and training smoothness while explicitly increasing the share of learning signal that comes from failures.
  arXiv  Detail & Related papers  (2025-12-22T16:34:21Z)
• Hard Samples, Bad Labels: Robust Loss Functions That Know When to Back Off [1.1582652820340928]
  Incorrectly labelled training data adversely affects the performance and generalizability of models trained through supervised learning.<n>We evaluate two novel loss functions, Blurry Loss and Piecewise-zero Loss, that enhance robustness to label errors by de-weighting or disregarding difficult-to-classify samples.
  arXiv  Detail & Related papers  (2025-11-20T16:30:55Z)
• Stress-Testing ML Pipelines with Adversarial Data Corruption [11.91482648083998]
  Regulators now demand evidence that high-stakes systems can withstand realistic, interdependent errors.<n>We introduce SAVAGE, a framework that formally models data-quality issues through dependency graphs and flexible corruption templates.<n>Savanage employs a bi-level optimization approach to efficiently identify vulnerable data subpopulations and fine-tune corruption severity.
  arXiv  Detail & Related papers  (2025-06-02T00:41:24Z)
• Bridging Internal Probability and Self-Consistency for Effective and Efficient LLM Reasoning [53.25336975467293]
  We present the first theoretical error decomposition analysis of methods such as perplexity and self-consistency.<n>Our analysis reveals a fundamental trade-off: perplexity methods suffer from substantial model error due to the absence of a proper consistency function.<n>We propose Reasoning-Pruning Perplexity Consistency (RPC), which integrates perplexity with self-consistency, and Reasoning Pruning, which eliminates low-probability reasoning paths.
  arXiv  Detail & Related papers  (2025-02-01T18:09:49Z)
• Fast and Accurate Error Simulation for CNNs against Soft Errors [64.54260986994163]
  We present a framework for the reliability analysis of Conal Neural Networks (CNNs) via an error simulation engine. These error models are defined based on the corruption patterns of the output of the CNN operators induced by faults. We show that our methodology achieves about 99% accuracy of the fault effects w.r.t. SASSIFI, and a speedup ranging from 44x up to 63x w.r.t.FI, that only implements a limited set of error models.
  arXiv  Detail & Related papers  (2022-06-04T19:45:02Z)
• Learn what you can't learn: Regularized Ensembles for Transductive Out-of-distribution Detection [76.39067237772286]
  We show that current out-of-distribution (OOD) detection algorithms for neural networks produce unsatisfactory results in a variety of OOD detection scenarios. This paper studies how such "hard" OOD scenarios can benefit from adjusting the detection method after observing a batch of the test data. We propose a novel method that uses an artificial labeling scheme for the test data and regularization to obtain ensembles of models that produce contradictory predictions only on the OOD samples in a test batch.
  arXiv  Detail & Related papers  (2020-12-10T16:55:13Z)
• FaultFace: Deep Convolutional Generative Adversarial Network (DCGAN) based Ball-Bearing Failure Detection Method [4.543665832042712]
  This paper proposes a methodology called FaultFace for failure detection on Ball-Bearing joints for rotational shafts. Deep Convolutional Generative Adversarial Network is employed to produce new faceportraits of the nominal and failure behaviors to get a balanced dataset.
  arXiv  Detail & Related papers  (2020-07-30T06:37:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.