Noisy Data is Destructive to Reinforcement Learning with Verifiable Rewards

• URL: http://arxiv.org/abs/2603.16140v1
• Date: Tue, 17 Mar 2026 05:48:32 GMT
• Title: Noisy Data is Destructive to Reinforcement Learning with Verifiable Rewards
• Abstract summary: Reinforcement learning with verifiable rewards (RLVR) has driven recent capability advances of large language models across various domains.<n>Recent studies suggest that improved RLVR algorithms allow models to learn effectively from incorrect annotations.<n>We show that these findings are invalid because the claimed 100% noisy training data is "contaminated" with clean data.
• Score: 9.797159765512236
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reinforcement learning with verifiable rewards (RLVR) has driven recent capability advances of large language models across various domains. Recent studies suggest that improved RLVR algorithms allow models to learn effectively from incorrect annotations, achieving performance comparable to learning from clean data. In this work, we show that these findings are invalid because the claimed 100% noisy training data is "contaminated" with clean data. After rectifying the dataset with a rigorous re-verification pipeline, we demonstrate that noise is destructive to RLVR. We show that existing RLVR algorithm improvements fail to mitigate the impact of noise, achieving similar performance to that of the basic GRPO. Furthermore, we find that the model trained on truly incorrect annotations performs 8-10% worse than the model trained on clean data across mathematical reasoning benchmarks. Finally, we show that these findings hold for real-world noise in Text2SQL tasks, where training on real-world, human annotation errors cause 5-12% lower accuracy than clean data. Our results show that current RLVR methods cannot yet compensate for poor data quality. High-quality data remains essential.

Related papers

• An Imperfect Verifier is Good Enough: Learning with Noisy Rewards [4.493881508510229]
  Reinforcement Learning with Verifiable Rewards (RLVR) has become a prominent method for post-training Large Language Models (LLMs)<n>The extent to which RLVR is robust to such noise and the verifier accuracy required for effective training remain unresolved questions.<n>We investigate these questions in the domains of code generation and scientific reasoning by introducing noise into RL training.
  arXiv  Detail & Related papers  (2026-04-09T00:15:01Z)
• Adaptive Ability Decomposing for Unlocking Large Reasoning Model Effective Reinforcement Learning [82.91265691530351]
  A$2$D is an Adaptive Ability Decomposing method for enhancing the effectiveness ofReinforcement Learning with verifiable rewards.<n>We first train a decomposer via RLVR without distillation, enabling it to decompose complex questions into a set of simpler sub-questions.<n>Next, we use this decomposer to annotate sub-questions for each question in the training dataset, and then train the reasoner under RLVR with sub-question guidance.
  arXiv  Detail & Related papers  (2026-01-31T14:48:23Z)
• The Role of Noisy Data in Improving CNN Robustness for Image Classification [0.0]
  This paper investigates the effect of deliberately introducing controlled noise into the training data to improve model robustness.<n>Experiments using a Resnet-18 model reveal that incorporating just 10% noisy data during training is sufficient to significantly reduce test loss and enhance accuracy under fully corrupted test conditions.
  arXiv  Detail & Related papers  (2026-01-12T22:26:24Z)
• Reasoning or Memorization? Unreliable Results of Reinforcement Learning Due to Data Contamination [67.67725938962798]
  Pre-training on massive web-scale corpora leaves Qwen2.5 susceptible to data contamination in widely used benchmarks.<n>We introduce a generator that creates fully clean arithmetic problems of arbitrary length and difficulty, dubbed RandomCalculation.<n>We show that only accurate reward signals yield steady improvements that surpass the base model's performance boundary.
  arXiv  Detail & Related papers  (2025-07-14T17:55:15Z)
• Reinforcement Learning for Reasoning in Large Language Models with One Training Example [117.86853102104256]
  We show that reinforcement learning with verifiable reward using one training example (1-shot RLVR) is effective in incentivizing the math reasoning capabilities of large language models (LLMs)<n>We identify some interesting phenomena during 1-shot RLVR, including cross-category generalization, increased frequency of self-reflection, and sustained test performance improvement.
  arXiv  Detail & Related papers  (2025-04-29T09:24:30Z)
• Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining [74.83412846804977]
  Reinforcement learning (RL)-based fine-tuning has become a crucial step in post-training language models.<n>We present a systematic end-to-end study of RL fine-tuning for mathematical reasoning by training models entirely from scratch.
  arXiv  Detail & Related papers  (2025-04-10T17:15:53Z)
• Impact of Noisy Supervision in Foundation Model Learning [91.56591923244943]
  This paper is the first work to comprehensively understand and analyze the nature of noise in pre-training datasets.<n>We propose a tuning method (NMTune) to affine the feature space to mitigate the malignant effect of noise and improve generalization.
  arXiv  Detail & Related papers  (2024-03-11T16:22:41Z)
• Understanding and Mitigating the Label Noise in Pre-training on Downstream Tasks [91.15120211190519]
  This paper aims to understand the nature of noise in pre-training datasets and to mitigate its impact on downstream tasks. We propose a light-weight black-box tuning method (NMTune) to affine the feature space to mitigate the malignant effect of noise.
  arXiv  Detail & Related papers  (2023-09-29T06:18:15Z)
• Double Descent and Overfitting under Noisy Inputs and Distribution Shift for Linear Denoisers [3.481985817302898]
  A concern about studying supervised denoising is that one might not always have noiseless training data from the test distribution. Motivated by this, we study supervised denoising and noisy-input regression under distribution shift.
  arXiv  Detail & Related papers  (2023-05-26T22:41:40Z)
• Improving the Robustness of Summarization Models by Detecting and Removing Input Noise [50.27105057899601]
  We present a large empirical study quantifying the sometimes severe loss in performance from different types of input noise for a range of datasets and model sizes. We propose a light-weight method for detecting and removing such noise in the input during model inference without requiring any training, auxiliary models, or even prior knowledge of the type of noise.
  arXiv  Detail & Related papers  (2022-12-20T00:33:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.