Detecting Data Contamination from Reinforcement Learning Post-training for Large Language Models

• URL: http://arxiv.org/abs/2510.09259v1
• Date: Fri, 10 Oct 2025 10:58:50 GMT
• Title: Detecting Data Contamination from Reinforcement Learning Post-training for Large Language Models
• Authors: Yongding Tao, Tian Wang, Yihong Dong, Huanyu Liu, Kechi Zhang, Xiaolong Hu, Ge Li,
• Abstract summary: Data contamination poses a significant threat to the reliable evaluation of Large Language Models (LLMs)<n>This issue arises when benchmark samples may inadvertently appear in training sets, compromising the validity of reported performance.<n>We propose Self-Critique as a specialized contamination detection method for RL post-training.
• Score: 30.267708813420587
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Data contamination poses a significant threat to the reliable evaluation of Large Language Models (LLMs). This issue arises when benchmark samples may inadvertently appear in training sets, compromising the validity of reported performance. While detection methods have been developed for the pre-training and Supervised Fine-Tuning stages, a critical research gap exists for the increasingly significant phase of Reinforcement Learning (RL) post-training. As RL post-training becomes pivotal for advancing LLM reasoning, the absence of specialized contamination detection methods in this paradigm presents a critical vulnerability. To address this, we conduct the first systematic study of data detection within RL post-training scenario and propose Self-Critique. Our method is motivated by a key observation: after RL phase, the output entropy distribution of LLMs tends to collapse into highly specific and sparse modes. Self-Critique probes for the underlying policy collapse, i.e., the model's convergence to a narrow reasoning path, which causes this entropy reduction. To facilitate this research, we also introduce RL-MIA, a benchmark constructed to simulate this specific contamination scenario. Extensive experiments show that Self-Critique significantly outperforms baseline methods across multiple models and contamination tasks, achieving an AUC improvement of up to 30%. Whereas existing methods are close to a random guess for RL-phase contamination, our method makes detection possible.

Related papers

• Data-regularized Reinforcement Learning for Diffusion Models at Scale [99.01056178660538]
  We introduce Data-regularized Diffusion Reinforcement Learning ( DDRL), a novel framework that uses the forward KL divergence to anchor the policy to an off-policy data distribution.<n>With over a million GPU hours of experiments and ten thousand double-blind evaluations, we demonstrate that DDRL significantly improves rewards while alleviating the reward hacking seen in RLs.
  arXiv  Detail & Related papers  (2025-12-03T23:45:07Z)
• Contamination Detection for VLMs using Multi-Modal Semantic Perturbation [73.76465227729818]
  Open-source Vision-Language Models (VLMs) have achieved state-of-the-art performance on benchmark tasks.<n>Pretraining corpora raise a critical concern for both practitioners and users: inflated performance due to test-set leakage.<n>We show that existing detection approaches either fail outright or exhibit inconsistent behavior.<n>We propose a novel simple yet effective detection method based on multi-modal semantic perturbation.
  arXiv  Detail & Related papers  (2025-11-05T18:59:52Z)
• On The Fragility of Benchmark Contamination Detection in Reasoning Models [20.455365567122985]
  Leaderboards for LRMs have turned evaluation into a competition, incentivizing developers to optimize directly on benchmark suites.<n>A shortcut to achieving higher rankings is to incorporate evaluation benchmarks into the training data, thereby yielding inflated performance, known as benchmark contamination.<n>We find that evading contamination detections for LRMs is alarmingly easy.
  arXiv  Detail & Related papers  (2025-09-30T21:40:54Z)
• Score-based Membership Inference on Diffusion Models [3.742113529511043]
  Membership inference attacks (MIAs) against diffusion models have emerged as a pressing privacy concern.<n>We present a theoretical and empirical study of score-based MIAs, focusing on the predicted noise vectors that diffusion models learn to approximate.<n>We show that the expected denoiser output points toward a kernel-weighted local mean of nearby training samples, such that its norm encodes proximity to the training set and thereby reveals membership.
  arXiv  Detail & Related papers  (2025-09-29T16:28:55Z)
• Mirage or Method? How Model-Task Alignment Induces Divergent RL Conclusions [22.83151273022573]
  counterintuitive phenomena have been reported in large language models (LLMs)<n>We identify a key factor that differentiates RL observations: whether the pretrained model already exhibits strong Model-Task Alignment.<n>Our findings show that while standard RL training remains consistently robust across settings, many of these counterintuitive results arise only when the model and task already exhibit strong model-task alignment.
  arXiv  Detail & Related papers  (2025-08-28T20:02:10Z)
• Training-Free Stein Diffusion Guidance: Posterior Correction for Sampling Beyond High-Density Regions [46.59494117137471]
  Training free diffusion guidance provides a flexible way to leverage off-the-shelf classifiers without additional training.<n>We introduce Stein Diffusion Guidance (SDG), a novel training-free framework grounded in a surrogate SOC objective.<n>Experiments on molecular low-density sampling tasks suggest that SDG consistently surpasses standard training-free guidance methods.
  arXiv  Detail & Related papers  (2025-07-07T21:14:27Z)
• Dissecting Long-Chain-of-Thought Reasoning Models: An Empirical Study [91.78803511141975]
  This work focuses on the roles of positive and negative samples in scaling reinforcement learning.<n>We identify substantial data inefficiency in group relative policy optimization, where over half of the samples yield zero advantage.<n>We investigate unstable performance across various reasoning models and benchmarks, attributing instability to uncertain problems with ambiguous outcomes.
  arXiv  Detail & Related papers  (2025-06-05T11:47:10Z)
• RN-F: A Novel Approach for Mitigating Contaminated Data in Large Language Models [0.8739101659113157]
  Residual-Noise Fingerprinting (RN-F) is a novel framework for detecting contaminated data in Large Language Models (LLMs)<n>RN-F is a single-pass, gradient-free detection method that leverages residual signal patterns without introducing additional floating-point operations.<n>We show that RN-F consistently outperforms existing state-of-the-art methods, achieving performance improvements of up to 10.5% in contamination detection metrics.
  arXiv  Detail & Related papers  (2025-05-19T15:32:49Z)
• Unconditional Truthfulness: Learning Unconditional Uncertainty of Large Language Models [104.55763564037831]
  We train a regression model that leverages attention maps, probabilities on the current generation step, and recurrently computed uncertainty scores from previously generated tokens.<n>Our evaluation shows that the proposed method is highly effective for selective generation, achieving substantial improvements over rivaling unsupervised and supervised approaches.
  arXiv  Detail & Related papers  (2024-08-20T09:42:26Z)
• Learning Off-policy with Model-based Intrinsic Motivation For Active Online Exploration [15.463313629574111]
  This paper investigates how to achieve sample-efficient exploration in continuous control tasks. We introduce an RL algorithm that incorporates a predictive model and off-policy learning elements. We derive an intrinsic reward without incurring parameters overhead.
  arXiv  Detail & Related papers  (2024-03-31T11:39:11Z)
• Take the Bull by the Horns: Hard Sample-Reweighted Continual Training Improves LLM Generalization [165.98557106089777]
  A key challenge is to enhance the capabilities of large language models (LLMs) amid a looming shortage of high-quality training data. Our study starts from an empirical strategy for the light continual training of LLMs using their original pre-training data sets. We then formalize this strategy into a principled framework of Instance-Reweighted Distributionally Robust Optimization.
  arXiv  Detail & Related papers  (2024-02-22T04:10:57Z)
• Reconstructing Graph Diffusion History from a Single Snapshot [87.20550495678907]
  We propose a novel barycenter formulation for reconstructing Diffusion history from A single SnapsHot (DASH) We prove that estimation error of diffusion parameters is unavoidable due to NP-hardness of diffusion parameter estimation. We also develop an effective solver named DIffusion hiTting Times with Optimal proposal (DITTO)
  arXiv  Detail & Related papers  (2023-06-01T09:39:32Z)
• To Repeat or Not To Repeat: Insights from Scaling LLM under Token-Crisis [50.31589712761807]
  Large language models (LLMs) are notoriously token-hungry during pre-training, and high-quality text data on the web is approaching its scaling limit for LLMs. We investigate the consequences of repeating pre-training data, revealing that the model is susceptible to overfitting. Second, we examine the key factors contributing to multi-epoch degradation, finding that significant factors include dataset size, model parameters, and training objectives.
  arXiv  Detail & Related papers  (2023-05-22T17:02:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.