An Imperfect Verifier is Good Enough: Learning with Noisy Rewards

• URL: http://arxiv.org/abs/2604.07666v1
• Date: Thu, 09 Apr 2026 00:15:01 GMT
• Title: An Imperfect Verifier is Good Enough: Learning with Noisy Rewards
• Abstract summary: Reinforcement Learning with Verifiable Rewards (RLVR) has become a prominent method for post-training Large Language Models (LLMs)<n>The extent to which RLVR is robust to such noise and the verifier accuracy required for effective training remain unresolved questions.<n>We investigate these questions in the domains of code generation and scientific reasoning by introducing noise into RL training.
• Score: 4.493881508510229
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) has become a prominent method for post-training Large Language Models (LLMs). However, verifiers are rarely error-free; even deterministic checks can be inaccurate, and the growing dependence on model-based judges exacerbates the issue. The extent to which RLVR is robust to such noise and the verifier accuracy required for effective training remain unresolved questions. We investigate these questions in the domains of code generation and scientific reasoning by introducing noise into RL training. Noise rates up to 15% yield peak validation accuracy within 2 percentage points of the clean baseline. These findings are consistent across controlled and model-based noise types, three model families (Qwen3, GLM4, Llama 3.1), and model sizes from 4B to 9B. Overall, the results indicate that imperfect verification does not constitute a fundamental barrier to RLVR. Furthermore, our findings suggest that practitioners should prioritize moderate accuracy with high precision over perfect verification.

Related papers

• Quantifying Empirical Compute-Supervision Tradeoffs in RLVR [0.15845117761091052]
  Recent theoretical work predicts that verifier noise affects the rate of learning but not its final outcome.<n>We test this prediction empirically by post-training Qwen2.5 (0.5B, 1.5B) with GRPO on GSM8K.<n>We find a structural asymmetry where false negatives monotonically degrade performance more quickly than false positives.
  arXiv  Detail & Related papers  (2026-05-24T20:28:12Z)
• Correct Answers from Sound Reasoning: Verifiable Process Supervision for Language Models [94.68358825189738]
  We propose a post-training framework for verifiable domains that jointly optimize prediction accuracy and reasoning quality.<n>We evaluate VPS on chess, a controlled testbed where reasoning steps can be deterministically verified against engine signals.<n>VPS preserves accuracy while significantly improving reasoning quality, reducing win-rate error by up to 30% and restoring consistency to near saturation.
  arXiv  Detail & Related papers  (2026-04-03T15:19:46Z)
• Noisy Data is Destructive to Reinforcement Learning with Verifiable Rewards [9.797159765512236]
  Reinforcement learning with verifiable rewards (RLVR) has driven recent capability advances of large language models across various domains.<n>Recent studies suggest that improved RLVR algorithms allow models to learn effectively from incorrect annotations.<n>We show that these findings are invalid because the claimed 100% noisy training data is "contaminated" with clean data.
  arXiv  Detail & Related papers  (2026-03-17T05:48:32Z)
• PRIME: A Process-Outcome Alignment Benchmark for Verifiable Reasoning in Mathematics and Engineering [71.15346406323827]
  We introduce PRIME, a benchmark for evaluating verifiers on Process-Outcome Alignment verification.<n>We find that current verifiers frequently fail to detect derivation flaws.<n>We propose a process-aware RLVR training paradigm utilizing verifiers selected via PRIME.
  arXiv  Detail & Related papers  (2026-02-12T04:45:01Z)
• JudgeRLVR: Judge First, Generate Second for Efficient Reasoning [20.448286296459344]
  Reinforcement Learning with Verifiable Rewards (RLVR) has become a standard paradigm for reasoning in Large Language Models.<n>In this paper, we argue that discriminative capability is a prerequisite for efficient generation.<n>We propose JudgeRLVR, a two-stage judge-then-generate paradigm.
  arXiv  Detail & Related papers  (2026-01-13T11:47:42Z)
• Mitigating LLM Hallucination via Behaviorally Calibrated Reinforcement Learning [32.32593439144886]
  Behavior-calibrated reinforcement learning allows smaller models to surpass frontier models in uncertainty quantification.<n>Our model's log-scale Accuracy-to-Hallucination Ratio gain (0.806) exceeds GPT-5's (0.207) in a challenging in-domain evaluation.
  arXiv  Detail & Related papers  (2025-12-22T22:51:48Z)
• Reinforcement Learning with Verifiable yet Noisy Rewards under Imperfect Verifiers [90.50039419576807]
  Reinforcement Learning with Verifiable Rewards (RLVR) trains policies against automated verifiers to avoid costly human labeling.<n>To reduce vulnerability to verifier hacking, many RLVR systems collapse rewards to binary $0,1$ during training.<n>This choice carries a cost: it introduces textitfalse negatives (rejecting correct answers, FNs) and textitfalse positives (accepting incorrect ones, FPs)
  arXiv  Detail & Related papers  (2025-10-01T13:56:44Z)
• Critique to Verify: Accurate and Honest Test-Time Scaling with RL-Trained Verifiers [63.99316853136304]
  Mirror-Critique is a framework that trains a verifier with informative critiques.<n>We deploy a small instruction-tuned model to synthesize high-quality critique data.<n>The resulting Mirror-Verifier is deployed to evaluate candidate solutions by generating multiple critiques per solution.
  arXiv  Detail & Related papers  (2025-09-27T06:50:24Z)
• Position: The Hidden Costs and Measurement Gaps of Reinforcement Learning with Verifiable Rewards [110.47944672736723]
  We argue that gains are often overstated due to three forces - an RLVR tax, evaluation pitfalls, and data contamination.<n>We propose a tax-aware training and evaluation protocol that co-optimizes accuracy, grounding, and calibrated abstention.
  arXiv  Detail & Related papers  (2025-09-26T05:06:25Z)
• Beyond Binary Rewards: Training LMs to Reason About Their Uncertainty [59.97939500426759]
  This paper describes RLCR, an approach to training reasoning models that jointly improves accuracy and confidence estimation.<n>We show that across diverse datasets, RLCR substantially improves calibration with no loss in accuracy.<n>We also demonstrate that verbalized confidence can be leveraged at test time to improve accuracy and calibration.
  arXiv  Detail & Related papers  (2025-07-22T17:56:01Z)
• From Accuracy to Robustness: A Study of Rule- and Model-based Verifiers in Mathematical Reasoning [41.02508512078575]
  We take mathematical reasoning as a case study and conduct a comprehensive analysis of various verifiers in both static evaluation and RL training scenarios.<n>First, we find that current open-source rule-based verifiers often fail to recognize equivalent answers presented in different formats, resulting in non-negligible false negative rates.<n>We investigate model-based verifiers as a potential solution to address these limitations.
  arXiv  Detail & Related papers  (2025-05-28T10:28:41Z)
• Reinforcement Learning for Reasoning in Large Language Models with One Training Example [117.86853102104256]
  We show that reinforcement learning with verifiable reward using one training example (1-shot RLVR) is effective in incentivizing the math reasoning capabilities of large language models (LLMs)<n>We identify some interesting phenomena during 1-shot RLVR, including cross-category generalization, increased frequency of self-reflection, and sustained test performance improvement.
  arXiv  Detail & Related papers  (2025-04-29T09:24:30Z)
• Robustness of Accuracy Metric and its Inspirations in Learning with Noisy Labels [51.66448070984615]
  We show that maximizing training accuracy on sufficiently many noisy samples yields an approximately optimal classifier. For validation, we prove that a noisy validation set is reliable, addressing the critical demand of model selection. We show characterizations of models trained with noisy labels, motivated by our theoretical results, and verify the utility of a noisy validation set.
  arXiv  Detail & Related papers  (2020-12-08T03:37:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.