DVD: A Robust Method for Detecting Variant Contamination in Large Language Model Evaluation

• URL: http://arxiv.org/abs/2601.04895v1
• Date: Thu, 08 Jan 2026 12:48:40 GMT
• Title: DVD: A Robust Method for Detecting Variant Contamination in Large Language Model Evaluation
• Authors: Renzhao Liang, Jingru Chen, Bo Jia, Bo Deng, Chenggang Xie, Yidong Wang, Ke Jin, Xin Wang, Linfeng Zhang, Cunxiang Wang,
• Abstract summary: textbfDVD is a single-sample detector that models the local output distribution induced by temperature sampling.<n>We construct the first benchmark for variant contamination across two domains Omni-MATH and SuperGPQA.<n>textbfDVD consistently outperforms perplexity-based, Min-$k$%++, edit-distance (CDD), and embedding-similarity baselines.
• Score: 24.086354908256293
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Evaluating large language models (LLMs) is increasingly confounded by \emph{variant contamination}: the training corpus contains semantically equivalent yet lexically or syntactically altered versions of test items. Unlike verbatim leakage, these paraphrased or structurally transformed variants evade existing detectors based on sampling consistency or perplexity, thereby inflating benchmark scores via memorization rather than genuine reasoning. We formalize this problem and introduce \textbf{DVD} (\textbf{D}etection via \textbf{V}ariance of generation \textbf{D}istribution), a single-sample detector that models the local output distribution induced by temperature sampling. Our key insight is that contaminated items trigger alternation between a \emph{memory-adherence} state and a \emph{perturbation-drift} state, yielding abnormally high variance in the synthetic difficulty of low-probability tokens; uncontaminated items remain in drift with comparatively smooth variance. We construct the first benchmark for variant contamination across two domains Omni-MATH and SuperGPQA by generating and filtering semantically equivalent variants, and simulate contamination via fine-tuning models of different scales and architectures (Qwen2.5 and Llama3.1). Across datasets and models, \textbf{DVD} consistently outperforms perplexity-based, Min-$k$\%++, edit-distance (CDD), and embedding-similarity baselines, while exhibiting strong robustness to hyperparameters. Our results establish variance of the generation distribution as a principled and practical fingerprint for detecting variant contamination in LLM evaluation.

Related papers

• Sharp Convergence Rates for Masked Diffusion Models [53.117058231393834]
  We develop a total-variation based analysis for the Euler method that overcomes limitations.<n>Our results relax assumptions on score estimation, improve parameter dependencies, and establish convergence guarantees.<n>Overall, our analysis introduces a direct TV-based error decomposition along the CTMC trajectory and a decoupling-based path-wise analysis for FHS.
  arXiv  Detail & Related papers  (2026-02-26T00:47:51Z)
• Beyond Raw Detection Scores: Markov-Informed Calibration for Boosting Machine-Generated Text Detection [105.14032334647932]
  Machine-generated texts (MGTs) pose risks such as disinformation and phishing, highlighting the need for reliable detection.<n> Metric-based methods, which extract statistically distinguishable features of MGTs, are often more practical than complex model-based methods that are prone to overfitting.<n>We propose a Markov-informed score calibration strategy that models two relationships of context detection scores that may aid calibration.
  arXiv  Detail & Related papers  (2026-02-08T16:06:12Z)
• Denoising diffusion networks for normative modeling in neuroimaging [1.0195618602298684]
  Most neuroimaging pipelines fit one model per imaging-derived phenotype (IDP)<n>We propose denoising diffusion probabilistic models (DDPMs) as a unified conditional density estimator for IDPs.<n>We evaluate on a synthetic benchmark with heteroscedastic and multimodal age effects and on UK Biobank FreeSurfer phenotypes, scaling from dimension of 2 to 200.
  arXiv  Detail & Related papers  (2026-01-24T06:19:10Z)
• Optimal Detection for Language Watermarks with Pseudorandom Collision [28.84134119819056]
  We introduce a statistical framework that captures structure through a hierarchical two-layer partition.<n>At its core is the concept of minimal units -- the smallest groups treatable as independent across units while permitting dependence within.<n>Applying to Gumbel-max and inverse-transform watermarks, our framework produces closed-form optimal rules.
  arXiv  Detail & Related papers  (2025-10-24T20:21:52Z)
• Efficient semantic uncertainty quantification in language models via diversity-steered sampling [46.23327887393273]
  We introduce a diversity-steered sampler that discourages semantically redundant outputs during decoding.<n>Key idea is to inject a continuous semantic-similarity penalty into the model's proposal distribution.<n>Being modular and requiring no gradient access to the base LLM, the framework promises to serve as a drop-in enhancement for uncertainty estimation.
  arXiv  Detail & Related papers  (2025-10-24T10:06:21Z)
• Vec2Summ: Text Summarization via Probabilistic Sentence Embeddings [2.2029818765681086]
  Vec2Summ represents a document collection using a single mean vector in the semantic embedding space.<n>We reconstruct fluent summaries using a generative language model.<n>Vec2Summ produces coherent summaries for topically focused, order-invariant corpora.
  arXiv  Detail & Related papers  (2025-08-09T15:31:02Z)
• Diversity by Design: Addressing Mode Collapse Improves scRNA-seq Perturbation Modeling on Well-Calibrated Metrics [1.756284691863048]
  Recent benchmarks reveal that models for single-cell perturbation response are often outperformed by simply predicting the dataset mean.<n>We trace this anomaly to a metric artifact: control-referenced deltas and unweighted error metrics reward mode collapse whenever the control is biased or the biological signal is sparse.<n>We introduce differentially expressed gene (DEG)-aware metrics, weighted mean-squared error (WMSE) and weighted delta $R2$ ($R2_w(Delta)$) with respect to all perturbations, that measure error in niche signals with high sensitivity.
  arXiv  Detail & Related papers  (2025-06-27T21:12:46Z)
• From Invariant Representations to Invariant Data: Provable Robustness to Spurious Correlations via Noisy Counterfactual Matching [14.679039741833305]
  Models that learn spurious correlations from training data often fail when deployed in new environments.<n>We propose a data-centric alternative that shifts the focus from learning invariant representations to leveraging invariant data pairs.<n>Noisy Counterfactual Matching improves robustness by leveraging even a small number of emphnoisy counterfactual pairs.
  arXiv  Detail & Related papers  (2025-05-30T17:42:32Z)
• Invariant Anomaly Detection under Distribution Shifts: A Causal Perspective [6.845698872290768]
  Anomaly detection (AD) is the machine learning task of identifying highly discrepant abnormal samples. Under the constraints of a distribution shift, the assumption that training samples and test samples are drawn from the same distribution breaks down. We attempt to increase the resilience of anomaly detection models to different kinds of distribution shifts.
  arXiv  Detail & Related papers  (2023-12-21T23:20:47Z)
• Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution [67.9215891673174]
  We propose score entropy as a novel loss that naturally extends score matching to discrete spaces. We test our Score Entropy Discrete Diffusion models on standard language modeling tasks.
  arXiv  Detail & Related papers  (2023-10-25T17:59:12Z)
• Tailoring Language Generation Models under Total Variation Distance [55.89964205594829]
  The standard paradigm of neural language generation adopts maximum likelihood estimation (MLE) as the optimizing method. We develop practical bounds to apply it to language generation. We introduce the TaiLr objective that balances the tradeoff of estimating TVD.
  arXiv  Detail & Related papers  (2023-02-26T16:32:52Z)
• A comprehensive comparative evaluation and analysis of Distributional Semantic Models [61.41800660636555]
  We perform a comprehensive evaluation of type distributional vectors, either produced by static DSMs or obtained by averaging the contextualized vectors generated by BERT. The results show that the alleged superiority of predict based models is more apparent than real, and surely not ubiquitous. We borrow from cognitive neuroscience the methodology of Representational Similarity Analysis (RSA) to inspect the semantic spaces generated by distributional models.
  arXiv  Detail & Related papers  (2021-05-20T15:18:06Z)
• Bridging the Gap Between Clean Data Training and Real-World Inference for Spoken Language Understanding [76.89426311082927]
  Existing models are trained on clean data, which causes a textitgap between clean data training and real-world inference. We propose a method from the perspective of domain adaptation, by which both high- and low-quality samples are embedding into similar vector space. Experiments on the widely-used dataset, Snips, and large scale in-house dataset (10 million training examples) demonstrate that this method not only outperforms the baseline models on real-world (noisy) corpus but also enhances the robustness, that is, it produces high-quality results under a noisy environment.
  arXiv  Detail & Related papers  (2021-04-13T17:54:33Z)
• On the Discrepancy between Density Estimation and Sequence Generation [92.70116082182076]
  log-likelihood is highly correlated with BLEU when we consider models within the same family. We observe no correlation between rankings of models across different families.
  arXiv  Detail & Related papers  (2020-02-17T20:13:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.