Benchmark Contamination: A Taxonomy Organized by Defeated Mitigation

• URL: http://arxiv.org/abs/2608.29463v1
• Date: Sat, 29 Aug 2026 23:04:09 GMT
• Title: Benchmark Contamination: A Taxonomy Organized by Defeated Mitigation
• Abstract summary: A benchmark score is a joint property of the model, the evaluation harness, the elicitation budget, the sampled population, and contamination status.<n>We introduce a taxonomy organized by the mitigation each type defeats.<n>Two variables fall below the prevalence-robust threshold registered in advance.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: A benchmark score is a joint property of the model, the evaluation harness, the elicitation budget, the sampled population, and contamination status. Leaderboards publish the model and the score, so capability and leakage stay observationally equivalent. Existing taxonomies classify contamination for automated detection, not the question a reporter faces at publication: given the mitigations already applied, which validity threats remain open? We introduce a taxonomy organized by the mitigation each type defeats -- direct, derivative, temporal, distributional, and acquired -- spanning training-time and evaluation-time leakage. Holding out a private test set closes the first alone. The fifth is acquired during the evaluation itself; because it is a property of one run, it must be recorded with the reported score rather than with the benchmark release. We operationalize it as a four-field disclosure protocol in which "unknown" is a valid entry, released under CC BY 4.0 with a JSON Schema, a validator, and worked examples. Two coders external to the design team applied a pre-registered instrument to 41 documents. Per-variable linear-weighted $κ$ runs from 0.00 to 0.35 (median 0.21) over 29 main-pass documents against a single-coder test-retest ceiling of 0.84, collapsing under the class skew the registration anticipated; pooling raises it to 0.46 through chance correction rather than better agreement. Two variables fall below the prevalence-robust threshold registered in advance: strata reporting and the acquired type introduced here. Disagreement concentrates on when a variable applies rather than on what a document states. Elicitation budgets are reported in 13% of documents, and no document addresses all five types. The contribution is the taxonomy, the score-side artifact that follows from it, and a pre-registered measurement of instrument reliability and current disclosure.

Related papers

• Admission Without Answers: Label-Free Certification and Experience Learning for LLM-Based Optimization Modeling [8.467936416935986]
  AdmitOR is an admission gate built on calibrated external behavioral evidence.<n>It is the first label-free admission mechanism designed around an explicitly calibrated false-discovery target.
  arXiv  Detail & Related papers  (2026-08-16T06:18:54Z)
• Looping Is Not Reliability: State-Bound Evidence and Typed Revision Contracts for Agentic Code Repair [36.56438114281786]
  We study the gap between finding a correct patch and retaining, verifying, and submitting it.<n>We derive an evidence-bound typed loop contract and instantiate its mechanically enforceable subset in a reference implementation.
  arXiv  Detail & Related papers  (2026-07-27T16:05:23Z)
• The C-index illusion: discrimination without calibration in published survival models [0.0]
  We reproduce three published survival-ML models across three structurally distinct domains.<n>We test five pre-registered hypotheses under a Holm-corrected family-wise error rate.
  arXiv  Detail & Related papers  (2026-07-21T19:13:54Z)
• Fantastic Adaptive Taxonomies and How to Use Them [100.20614173157708]
  AdaMAST converts execution traces into a compact, evidence-grounded failure taxonomy.<n>No code is hand-authored, no trace human-annotated.<n>In agent-system search, taxonomy-coded diagnoses of failed candidates outperform free-form.
  arXiv  Detail & Related papers  (2026-07-17T17:41:16Z)
• Evaluation Cards: An Interpretive Layer for AI Evaluation Reporting [44.70980205167796]
  AI evaluation results are produced at scale but reported inconsistently across leaderboards, model cards, benchmark papers, and company blogs.<n>Recent efforts address isolated components but leave three gaps: they cover only narrow slices of the evaluation lifecycle and do not compose into a single interpretable record.<n>We present EvalCards, an operational reporting layer that composes benchmark metadata, evaluation run data, and model metadata into a unified record.
  arXiv  Detail & Related papers  (2026-06-08T17:55:02Z)
• CoEval: Ranking Language Models for Custom Tasks Without Labeled Data or Trustworthy Benchmarks [47.027290803102666]
  We present CoEval, an open framework that supplies a trustworthy, task-specific signal through ensemble self-evaluation.<n>A pool of models rotates through all three roles, teacher, student, and judge, to generate a fresh, contamination-free benchmark.
  arXiv  Detail & Related papers  (2026-06-02T13:41:43Z)
• Let the Results Speak: A Replication-First Paradigm for LLM Behavioral Benchmarking [22.825786049667602]
  We propose a replication-first paradigm to anchor validity to a single human-rater consensus.<n>We certify the instrument via four properties -- reliability across K runs, cross-instrument replication across architecturally distinct judges, historical-footprint calibration via judges from earlier training cohorts, and pre-registered prediction.<n>We test it on emotional accompaniment by letting the self-evolve data-driven across: the dimensions are not pre-stipulated and the procedure stabilizes to a 9-dimension set.
  arXiv  Detail & Related papers  (2026-05-27T03:41:11Z)
• From Theory to Decision Rule: Calibrating the Noisy-Label Crossover for Vision-Language Model Weak Supervision Across Three Medical-Imaging Benchmarks [41.99844472131922]
  We provide a benchmark calibration for foundation-model weak labels.<n>The crossover predicted by theory appears at ng100 on PCAM, 20-50 on ISIC, and 250-500 on NIH-CXR.<n>A structured-vs-random noise sign flip on NIH-CXR shows that the rate-only formulation of the bound is incomplete.
  arXiv  Detail & Related papers  (2026-05-23T23:15:07Z)
• What Twelve LLM Agent Benchmark Papers Disclose About Themselves: A Pilot Audit and an Open Scoring Schema [0.0]
  We read twelve well-known LLM agent benchmark papers and recorded, dimension by dimension, what each paper actually says about how its evaluation was run.<n>We designed a small audit schema (five fields: benchmark identity, harness specification, inference settings, cost reporting, failure breakdown)<n>We score the disclosure of an agent run, not its correctness, and make no claim that disclosure implies a trustworthy result.
  arXiv  Detail & Related papers  (2026-05-20T17:02:36Z)
• When No Benchmark Exists: Validating Comparative LLM Safety Scoring Without Ground-Truth Labels [34.86529553336423]
  Many deployments must compare candidate language models for safety before a labeled benchmark exists for the relevant language, sector, or regulatory regime.<n>We formalize this setting as benchmarkless comparative safety scoring and specify the contract under which a scenario-based audit can be interpreted as deployment evidence.<n>Scores are valid only under a fixed scenario pack, rubric, auditor, judge, sampling configuration, and rerun budget.
  arXiv  Detail & Related papers  (2026-05-07T17:56:41Z)
• DWBench: Holistic Evaluation of Watermark for Dataset Copyright Auditing [43.881484429055654]
  dataset watermark technique holds promise for auditing and verifying usage.<n>We develop DWBench, a unified benchmark and open-source toolkit for systematically evaluating image dataset watermark techniques.<n>We present the results of two new metrics: sample significance for fine-grained watermark distinguishability and verification success rate for dataset-level auditing.
  arXiv  Detail & Related papers  (2026-02-14T01:09:19Z)
• SpatialBench-UC: Uncertainty-Aware Evaluation of Spatial Prompt Following in Text-to-Image Generation [0.0]
  SpatialBench-UC is a small, reproducible benchmark for pairwise spatial relations.<n>We release a benchmark package, versioned prompts, pinned configs, per-sample checker outputs, and report tables.<n>We evaluate three baselines, Stable Diffusion 1.5, SD 1.5 BoxDiff, and SD 1.4 GLIGEN.
  arXiv  Detail & Related papers  (2026-01-19T23:37:10Z)
• CertDW: Towards Certified Dataset Ownership Verification via Conformal Prediction [48.82467166657901]
  We propose the first certified dataset watermark (i.e., CertDW) and CertDW-based certified dataset ownership verification method.<n>Inspired by conformal prediction, we introduce two statistical measures, including principal probability (PP) and watermark robustness (WR)<n>We prove there exists a provable lower bound between PP and WR, enabling ownership verification when a suspicious model's WR value significantly exceeds the PP values of benign models trained on watermark-free datasets.
  arXiv  Detail & Related papers  (2025-06-16T07:17:23Z)
• Arbitrariness and Social Prediction: The Confounding Role of Variance in Fair Classification [31.392067805022414]
  Variance in predictions across different trained models is a significant, under-explored source of error in fair binary classification. In practice, the variance on some data examples is so large that decisions can be effectively arbitrary. We develop an ensembling algorithm that abstains from classification when a prediction would be arbitrary.
  arXiv  Detail & Related papers  (2023-01-27T06:52:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.