All Leaks Count, Some Count More: Interpretable Temporal Contamination Detection in LLM Backtesting

• URL: http://arxiv.org/abs/2602.17234v1
• Date: Thu, 19 Feb 2026 10:28:00 GMT
• Title: All Leaks Count, Some Count More: Interpretable Temporal Contamination Detection in LLM Backtesting
• Abstract summary: We introduce a claim-level framework for detecting and quantifying this emphtemporal knowledge leakage<n>Our approach decomposes model rationales into atomic claims and categorizes them by temporal verifiability, then applies textitShapley values to measure each claim's contribution to the prediction.<n>TimeSPEC reduces Shapley-DCLR while preserving task performance, demonstrating that explicit, interpretable claim-level verification outperforms prompt-based temporal constraints for reliable backtesting.
• Score: 5.719582970287213
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: To evaluate whether LLMs can accurately predict future events, we need the ability to \textit{backtest} them on events that have already resolved. This requires models to reason only with information available at a specified past date. Yet LLMs may inadvertently leak post-cutoff knowledge encoded during training, undermining the validity of retrospective evaluation. We introduce a claim-level framework for detecting and quantifying this \emph{temporal knowledge leakage}. Our approach decomposes model rationales into atomic claims and categorizes them by temporal verifiability, then applies \textit{Shapley values} to measure each claim's contribution to the prediction. This yields the \textbf{Shapley}-weighted \textbf{D}ecision-\textbf{C}ritical \textbf{L}eakage \textbf{R}ate (\textbf{Shapley-DCLR}), an interpretable metric that captures what fraction of decision-driving reasoning derives from leaked information. Building on this framework, we propose \textbf{Time}-\textbf{S}upervised \textbf{P}rediction with \textbf{E}xtracted \textbf{C}laims (\textbf{TimeSPEC}), which interleaves generation with claim verification and regeneration to proactively filter temporal contamination -- producing predictions where every supporting claim can be traced to sources available before the cutoff date. Experiments on 350 instances spanning U.S. Supreme Court case prediction, NBA salary estimation, and stock return ranking reveal substantial leakage in standard prompting baselines. TimeSPEC reduces Shapley-DCLR while preserving task performance, demonstrating that explicit, interpretable claim-level verification outperforms prompt-based temporal constraints for reliable backtesting.

Related papers

• Zero Gap Is Not Restoration: Stratified Per-Question Probability Evaluation and Step-wise Mitigation of Benchmark Contamination [8.221134173409895]
  We propose textbfSA-AP (textbfStratified textbfAggregate of textbfPer-question textbfProbability textbfGaps): estimate each question's solve probability by sampling, difference it against the clean model per question, and aggregate within groups defined by the clean model's solve probability.
  arXiv  Detail & Related papers  (2026-08-07T15:37:03Z)
• Capacity-Constrained Online Convex Optimization with Delayed Feedback [58.59385794080679]
  We study delayed online convex optimization (OCO) under a hard capacity constraint.<n>We find that for first-order feedback, capacity $C = (log T)$ suffices to recover standard delayed OCO rates up to logarithmic factors.<n>For bandit feedback, the regret rates are modulated by powers of $(1 + _textmax/C)$, where $_textmax$ is the maximum number of pending observations at any time.
  arXiv  Detail & Related papers  (2026-06-10T06:37:06Z)
• Teaching Large Language Models When Not to Know: Learning Temporal Critique for Ex-Ante Reasoning [21.280999165642932]
  Large language models (LLMs) often fail to reason under temporal cutoffs.<n>We propose TCFT, a framework that trains models to acquire cutoff-aware temporal verification.
  arXiv  Detail & Related papers  (2026-05-14T09:49:35Z)
• OracleProto: A Reproducible Framework for Benchmarking LLM Native Forecasting via Knowledge Cutoff and Temporal Masking [17.407881336351597]
  OracleProto is a framework for evaluating large language models' native forecasting capability.<n>It reconstructs resolved events into time-bounded forecasting samples by combining model-cutoff-aligned sample admission, tool-level temporal masking, content-level leakage detection, discrete answer normalization, and hierarchical scoring.<n>It provides a unified interface for fair cross-model comparison and a controlled signal source for downstream SFT and RL.
  arXiv  Detail & Related papers  (2026-05-05T13:50:50Z)
• Teaching LLMs to Ask: Self-Querying Category-Theoretic Planning for Under-Specified Reasoning [1.8055130471307603]
  Inference-time planning with large language models frequently breaks under partial observability.<n>We introduce textbfSelf-Querying Bidirectional Categorical Planning (SQ-BCP), which explicitly represents precondition status.<n>We prove that when the verifier succeeds and hard constraints pass deterministic checks, accepted plans are compatible with goal requirements.
  arXiv  Detail & Related papers  (2026-01-27T19:41:10Z)
• Defenses Against Prompt Attacks Learn Surface Heuristics [40.392588465939106]
  Large language models (LLMs) are increasingly deployed in security-sensitive applications.<n>LLMs may override intended logic when adversarial instructions appear in user queries or retrieved content.<n>Recent defenses rely on supervised fine-tuning with benign and malicious labels.
  arXiv  Detail & Related papers  (2026-01-12T04:12:48Z)
• LLMLagBench: Identifying Temporal Training Boundaries in Large Language Models [0.0]
  Large Language Models (LLMs) are pretrained on textual data up to a specific temporal cutoff.<n>LLMs may inadvertently blend outdated time-sensitive information with general knowledge during reasoning tasks.
  arXiv  Detail & Related papers  (2025-11-15T09:08:10Z)
• DistDF: Time-Series Forecasting Needs Joint-Distribution Wasserstein Alignment [92.70019102733453]
  Training time-series forecast models requires aligning the conditional distribution of model forecasts with that of the label sequence.<n>We propose DistDF, which achieves alignment by alternatively minimizing a discrepancy between the conditional forecast and label distributions.
  arXiv  Detail & Related papers  (2025-10-28T16:09:59Z)
• ResCP: Reservoir Conformal Prediction for Time Series Forecasting [39.81023599249223]
  Conformal prediction offers a powerful framework for building distribution-free prediction intervals for exchangeable data.<n>We propose Reservoir Conformal Prediction (ResCP), a novel training-free conformal prediction method for time series.
  arXiv  Detail & Related papers  (2025-10-06T17:37:44Z)
• Reinforcement Learning with Verifiable yet Noisy Rewards under Imperfect Verifiers [90.50039419576807]
  Reinforcement Learning with Verifiable Rewards (RLVR) trains policies against automated verifiers to avoid costly human labeling.<n>To reduce vulnerability to verifier hacking, many RLVR systems collapse rewards to binary $0,1$ during training.<n>This choice carries a cost: it introduces textitfalse negatives (rejecting correct answers, FNs) and textitfalse positives (accepting incorrect ones, FPs)
  arXiv  Detail & Related papers  (2025-10-01T13:56:44Z)
• Can Prompts Rewind Time for LLMs? Evaluating the Effectiveness of Prompted Knowledge Cutoffs [31.64130018833542]
  Large Language Models (LLMs) are widely used for temporal prediction, but their reliance on pretraining data raises contamination concerns.<n>We investigate the capability of prompting to simulate earlier knowledge cutoff in LLMs.<n>Results demonstrate that while prompt-based simulated knowledge cutoffs show effectiveness when directly queried with the information after that date, they struggle to induce forgetting when the forgotten content is not directly asked but causally related to the query.
  arXiv  Detail & Related papers  (2025-09-26T20:37:44Z)
• COIN: Uncertainty-Guarding Selective Question Answering for Foundation Models with Provable Risk Guarantees [51.5976496056012]
  COIN is an uncertainty-guarding selection framework that calibrates statistically valid thresholds to filter a single generated answer per question.<n>COIN estimates the empirical error rate on a calibration set and applies confidence interval methods to establish a high-probability upper bound on the true error rate.<n>We demonstrate COIN's robustness in risk control, strong test-time power in retaining admissible answers, and predictive efficiency under limited calibration data.
  arXiv  Detail & Related papers  (2025-06-25T07:04:49Z)
• Language Bottleneck Models: A Framework for Interpretable Knowledge Tracing and Beyond [55.984684518346924]
  We recast Knowledge Tracing as an inverse problem: learning the minimum natural-language summary that makes past answers explainable and future answers predictable.<n>Our Language Bottleneck Model (LBM) consists of an encoder LLM that writes an interpretable knowledge summary and a frozen decoder LLM that must reconstruct and predict student responses using only that summary text.<n> Experiments on synthetic arithmetic benchmarks and the large-scale Eedi dataset show that LBMs rival the accuracy of state-of-the-art KT and direct LLM methods while requiring orders-of-magnitude fewer student trajectories.
  arXiv  Detail & Related papers  (2025-06-20T13:21:14Z)
• ExAnte: A Benchmark for Ex-Ante Inference in Large Language Models [12.948099229475265]
  Large language models (LLMs) face significant challenges in ex-ante reasoning.<n>Even with explicit prompts enforcing temporal cutoffs, LLMs often generate outputs influenced by internalized knowledge of events beyond the specified cutoff.<n>This paper introduces a novel task and benchmark designed to evaluate the ability of LLMs to reason while adhering to such temporal constraints.
  arXiv  Detail & Related papers  (2025-05-26T05:39:57Z)
• Decision from Suboptimal Classifiers: Excess Risk Pre- and Post-Calibration [52.70324949884702]
  We quantify the excess risk incurred using approximate posterior probabilities in batch binary decision-making.<n>We identify regimes where recalibration alone addresses most of the regret, and regimes where the regret is dominated by the grouping loss.<n>On NLP experiments, we show that these quantities identify when the expected gain of more advanced post-training is worth the operational cost.
  arXiv  Detail & Related papers  (2025-03-23T10:52:36Z)
• Fact-Checking the Output of Large Language Models via Token-Level Uncertainty Quantification [116.77055746066375]
  Large language models (LLMs) are notorious for hallucinating, i.e., producing erroneous claims in their output. We propose a novel fact-checking and hallucination detection pipeline based on token-level uncertainty quantification.
  arXiv  Detail & Related papers  (2024-03-07T17:44:17Z)
• Mitigating Temporal Misalignment by Discarding Outdated Facts [58.620269228776294]
  Large language models are often used under temporal misalignment, tasked with answering questions about the present. We propose fact duration prediction: the task of predicting how long a given fact will remain true. Our data and code are released publicly at https://github.com/mikejqzhang/mitigating_misalignment.
  arXiv  Detail & Related papers  (2023-05-24T07:30:08Z)
• Extracting or Guessing? Improving Faithfulness of Event Temporal Relation Extraction [87.04153383938969]
  We improve the faithfulness of TempRel extraction models from two perspectives. The first perspective is to extract genuinely based on contextual description. The second perspective is to provide proper uncertainty estimation.
  arXiv  Detail & Related papers  (2022-10-10T19:53:13Z)
• Retrieval Based Time Series Forecasting [37.48394754614059]
  Time series data appears in a variety of applications such as smart transportation and environmental monitoring. One of the fundamental problems for time series analysis is time series forecasting. We show both theoretically and empirically that the uncertainty could be effectively reduced by retrieving relevant time series as references.
  arXiv  Detail & Related papers  (2022-09-27T16:43:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.