When Is Benchmark Contamination Detectable? Information Limits and Power-Calibrated Audits

• URL: http://arxiv.org/abs/2608.07914v1
• Date: Sat, 08 Aug 2026 04:41:36 GMT
• Title: When Is Benchmark Contamination Detectable? Information Limits and Power-Calibrated Audits
• Abstract summary: Behavioral contamination detectors can return "no evidence" either because a benchmark is clean or because the audit has little power.<n>We formalize this distinction for a benchmark in which an unknown fraction alpha of items was seen during training.<n>Our empirical finding is two-sided. Frozen calibration efficacy predicts held-out power curves, with R2 = 0.83-0.98 across six exact-permutation channels.<n>A predeclared two-stage planner that simulates the deployed test repairs the budgets, is uniformly conservative, and abstains when its probe does not transport.
• Score: 6.908972852063454
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Behavioral contamination detectors can return "no evidence" either because a benchmark is clean or because the audit has little power. We formalize this distinction for a benchmark in which an unknown fraction alpha of items was seen during training. With matched clean and seen controls, the behavioral channel is the sparse mixture Q_alpha = (1 - alpha) P_0 + alpha P_1, and an exact second-moment argument shows that detectability is governed by alpha * rho * sqrt(m), where rho^2 = chi^2(P_1 || P_0) measures behavioral separability. Any scalar detector reduces to its efficacy, ef = |E_1 f - E_0 f| / sqrt(Var_0(f)) <= rho, which can be estimated from controls before the audit is run. A separate sample-split certificate lower-bounds alpha distribution-free, without requiring an orientation assumption. Our empirical finding is two-sided. Frozen calibration efficacy predicts held-out power curves, with R^2 = 0.83-0.98 across six exact-permutation channels, but the efficacy-only Gaussian budget is miscalibrated at the small sample sizes it prescribes, failing in 9/9 gate-passing channels even though efficacy itself transports. The failure is in the inversion, not the calibration. A predeclared two-stage planner that simulates the deployed test repairs the budgets, is uniformly conservative, and abstains when its probe does not transport. The certificate is valid but vacuous at audit scale, and a five-seed paired injection study recovers the mechanism ordering verbatim > paraphrase > surface, in which the apparent answer-only signal is explained by baseline drift. We report the audit contract and its failures together: a non-rejection is interpretable only alongside the efficacy, budget, and validity gates that produced it.

Related papers

• Certifying bipartite entanglement on a superconducting processor from a corrected QAOA cost layer [0.0]
  We report a pre-registered single-device study on the nine-qubit superconducting processor Red at the Barcelona Supercomputing Center.<n>We certify bipartite entanglement of a corrected QAOA cost layer on the good coupler at depth one by two-qubit tomographic negativity.<n>We measure the device physics directly: in a pre-registered manipulation on a fixed Bell preparation, reducing the CZ pulse amplitude causally reduces the certified negativity.
  arXiv  Detail & Related papers  (2026-09-08T22:23:35Z)
• Excess Separability: Nuisance-Controlled Residual-Stream Probing for Benchmark Contamination Detection [0.0]
  Benchmark contamination is diagnosed with n-gram overlap, likelihood-based membership inference, or canary strings.<n>We show the natural way to do this does not work, specify one that survives measurement, then find that the correction making it work carries more variance than the null it is tested against.
  arXiv  Detail & Related papers  (2026-08-12T23:27:20Z)
• Post-Hoc Trajectory-Risk Certification for Modular LLM-Based Security Agents [1.4915002678801434]
  We show that Bonferroni allocation is distribution-free but conservative under correlated errors.<n> coarse-to-fine label selection can create near-perfect measured correlation without learned dependence.<n>A modular certificate using per-stage certificates and a pairwise overlap bound yields a positive average gain of 0.6%.
  arXiv  Detail & Related papers  (2026-08-04T23:48:36Z)
• A Reference-Free Score for Detecting Silent Reasoning Failures in Large Language Models [1.0602247913671219]
  Reasoning Answer Faithfulness Score (RAFS) is a reference free, instance level diagnostic of whether an emitted mathematical trace is locally credible.<n>RAFS combines step validity, reasoning to answer entailment and counterfactual sensitivity, answer consensus, and conditional reasoning stability.<n>RAFS is intended to complement mathematical answer accuracy with an auditable warning signal for silent reasoning failures and answer extraction errors.
  arXiv  Detail & Related papers  (2026-07-28T07:28:38Z)
• Zero-Flow Two-Sample Tests [47.04254380987003]
  We propose a new approach to two-sample testing for deciding whether two sets of samples are drawn from the same distribution.<n>The test is built on a statistical discrepancy based on the zero-flow criterion, termed zero-flow discrepancy (ZFD)<n>We prove the validity of ZFD and propose a practical testing procedure, termed the zero-flow two-sample test (ZF2ST)
  arXiv  Detail & Related papers  (2026-07-23T17:27:23Z)
• Self-Poisoning in Adaptive Out-of-Distribution Detection: A Sharp-Threshold Theory and Certified Label-Free Calibration [0.0]
  Test-time adaptive out-of-distribution detectors update a memory bank from the unlabelled stream.<n>We show this adaptation obeys a provable dynamical law.<n>For the complementary static-calibration failure under drift we give CDC, which restores nominal FPR label-free on all tested drift-affected cells.
  arXiv  Detail & Related papers  (2026-07-23T09:43:46Z)
• Operational Proto-Introspection in Looped Language Models: Process-Quality Taps, Executable Branching, and the Readout-Control Boundary [0.0]
  Can a language model read the quality of its ongoing computation?<n>Can an external intervention turn that readout into better outcomes?<n>We test both questions in a frozen 2.6B looped transformer, Ouro-RLTT.
  arXiv  Detail & Related papers  (2026-07-20T22:40:36Z)
• What Accuracy and Gradient Cosine Miss: Evaluating Feedback Alignment via Scale Stability, Reference Validity, and Depth Utility [48.10132234701036]
  We propose a diagnostic evaluation protocol based on three checks -- scale stability, reference validity, and depth utility -- together with per-layer rather than aggregate cosine reporting.<n>Across multiple architectures and methods, our protocol identifies all failures with wide calibration margins.
  arXiv  Detail & Related papers  (2026-06-19T06:04:17Z)
• Demystifying Variance in Circuit Discovery of LLMs [51.97489679448601]
  We develop a new circuit discovery method that substantially lessens resampling variance.<n>We show that rephrasing variance arises because prompts with different templates tend to activate different circuits in the model.<n>We argue that sparsity, which has been claimed to form more compact and interpretable task circuits, fails to solve this problem.
  arXiv  Detail & Related papers  (2026-06-15T16:25:07Z)
• Finite Resources False Discovery Rate Control in Structured Hypothesis Spaces [47.16593717985932]
  We present a framework for controlling the false discovery rate when each hypothesis is evidenced only by a finite count of null draws.<n>We present two decision rules that are both robust to structural mis-specification, yet offer a distinct trade-off between exact FDR control and statistical power.
  arXiv  Detail & Related papers  (2026-06-13T16:56:52Z)
• Conformal calibration and look-elsewhere effect in anomaly detection for new-physics searches [0.0]
  Machine-learned anomaly detection is reshaping searches for new physics, but it has outrun the statistics used to interpret it.<n>We propose a calibration layer, built on conformal prediction, that turns any anomaly score into a defensible significance.
  arXiv  Detail & Related papers  (2026-06-11T18:00:02Z)
• Correction and Corruption: A Two-Rate View of Error Flow in LLM Protocols [51.56484100374058]
  We propose a paired-outcome measurement interface for auditing a single protocol step on exact-match tasks.<n>For each instance, the interface records a baseline correctness bit and a post-step correctness bit.<n>These rates predict accuracy changes and define a reusable empirical interface testable across seeds, mixtures, and pipelines.
  arXiv  Detail & Related papers  (2026-04-20T13:25:40Z)
• Detecting Unobserved Confounders: A Kernelized Regression Approach [46.52607207396279]
  Kernel Regression Confounder Detection (KRCD) is a novel method for detecting unobserved confounding in nonlinear observational data under single-environment conditions.<n>Test statistic whose significant deviation from zero indicates unobserved confounding.<n>Experiments on synthetic benchmarks and the Twins dataset demonstrate that KRCD not only outperforms existing baselines but also achieves superior computational efficiency.
  arXiv  Detail & Related papers  (2026-01-01T04:26:02Z)
• Reinforcement Learning with Verifiable yet Noisy Rewards under Imperfect Verifiers [90.50039419576807]
  Reinforcement Learning with Verifiable Rewards (RLVR) trains policies against automated verifiers to avoid costly human labeling.<n>To reduce vulnerability to verifier hacking, many RLVR systems collapse rewards to binary $0,1$ during training.<n>This choice carries a cost: it introduces textitfalse negatives (rejecting correct answers, FNs) and textitfalse positives (accepting incorrect ones, FPs)
  arXiv  Detail & Related papers  (2025-10-01T13:56:44Z)
• Tracking the risk of a deployed model and detecting harmful distribution shifts [105.27463615756733]
  In practice, it may make sense to ignore benign shifts, under which the performance of a deployed model does not degrade substantially. We argue that a sensible method for firing off a warning has to both (a) detect harmful shifts while ignoring benign ones, and (b) allow continuous monitoring of model performance without increasing the false alarm rate.
  arXiv  Detail & Related papers  (2021-10-12T17:21:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.