Testing For Distribution Shifts with Conditional Conformal Test Martingales

• URL: http://arxiv.org/abs/2602.13848v1
• Date: Sat, 14 Feb 2026 18:47:26 GMT
• Title: Testing For Distribution Shifts with Conditional Conformal Test Martingales
• Abstract summary: We propose a sequential test for detecting arbitrary distribution shifts that allows conformal test martingales (CTMs) to work under a fixed, reference-conditional setting.<n>Existing CTM detectors construct test martingales by continually growing a reference set with each incoming sample, using it to assess how atypical the new sample is relative to past observations.<n>While this design yields anytime-valid type-I error control, it suffers from test-time contamination: after a change, post-shift observations enter the reference set and dilute the evidence for distribution shift, increasing detection delay and reducing power.<n>Our method detects shifts faster than standard
• Score: 20.369245689839822
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: We propose a sequential test for detecting arbitrary distribution shifts that allows conformal test martingales (CTMs) to work under a fixed, reference-conditional setting. Existing CTM detectors construct test martingales by continually growing a reference set with each incoming sample, using it to assess how atypical the new sample is relative to past observations. While this design yields anytime-valid type-I error control, it suffers from test-time contamination: after a change, post-shift observations enter the reference set and dilute the evidence for distribution shift, increasing detection delay and reducing power. In contrast, our method avoids contamination by design by comparing each new sample to a fixed null reference dataset. Our main technical contribution is a robust martingale construction that remains valid conditional on the null reference data, achieved by explicitly accounting for the estimation error in the reference distribution induced by the finite reference set. This yields anytime-valid type-I error control together with guarantees of asymptotic power one and bounded expected detection delay. Empirically, our method detects shifts faster than standard CTMs, providing a powerful and reliable distribution-shift detector.

Related papers

• CITE: Anytime-Valid Statistical Inference in LLM Self-Consistency [10.34950275095264]
  We study anytime-valid certification of a prespecified target answer as unique mode of the model's response distribution.<n>We propose the Certification by Intersection-union Testing with E-processes (CITE) algorithm, which provably controls false certification at any prescribed level.<n>We also prove an category-set-size-free stopping-time rate, establish matching minimax lower bounds up to constants in the main regime, and extend the construction to confidence-weighted voting.
  arXiv  Detail & Related papers  (2026-05-07T08:41:51Z)
• Distributional Spectral Diagnostics for Localizing Grokking Transitions [4.480801209867367]
  In grokking, a model first fits the training data while test accuracy remains low, and only later begins to generalize.<n>We ask whether this transition can be localized from observed training trajectories before the test accuracy rises, and formulate grokking transition localization as a diagnostic problem with an explicit threshold/FPR/lead-time trade-off.
  arXiv  Detail & Related papers  (2026-05-07T03:08:58Z)
• Data Fusion with Distributional Equivalence Test-then-pool [5.5945020319921435]
  Existing test-then-pool (TTP) procedures address this concern by testing for equality of control outcomes between historical and concurrent trials.<n>We develop a new TTP framework that fuses control arms while rigorously controlling the Type-I error rate of the final treatment effect test.
  arXiv  Detail & Related papers  (2026-03-12T12:38:35Z)
• Technical note on Sequential Test-Time Adaptation via Martingale-Driven Fisher Prompting [3.5808917363708743]
  M-FISHER is a method for sequential distribution shift detection and stable adaptation in streaming data.<n>For detection, we construct an exponential martingale from non-conformity scores and apply Ville's inequality to obtain time-uniform guarantees on false alarm control.<n>For adaptation, we show that Fisher-preconditioned updates of prompt parameters implement natural gradient descent on the distributional manifold.
  arXiv  Detail & Related papers  (2025-10-04T15:31:26Z)
• BayesTTA: Continual-Temporal Test-Time Adaptation for Vision-Language Models via Gaussian Discriminant Analysis [41.09181390655176]
  Vision-language models (VLMs) such as CLIP achieve strong zero-shot recognition but degrade significantly under textittemporally evolving distribution shifts common in real-world scenarios.<n>We formalize this practical problem as textitContinual-Temporal Test-Time Adaptation (CT-TTA), where test distributions evolve gradually over time.<n>We propose textitBayesTTA, a Bayesian adaptation framework that enforces temporally consistent predictions and dynamically aligns visual representations.
  arXiv  Detail & Related papers  (2025-07-11T14:02:54Z)
• A Sample Efficient Conditional Independence Test in the Presence of Discretization [54.047334792855345]
  Conditional Independence (CI) tests directly to discretized data can lead to incorrect conclusions.<n>Recent advancements have sought to infer the correct CI relationship between the latent variables through binarizing observed data.<n>Motivated by this, this paper introduces a sample-efficient CI test that does not rely on the binarization process.
  arXiv  Detail & Related papers  (2025-06-10T12:41:26Z)
• Error-quantified Conformal Inference for Time Series [55.11926160774831]
  Uncertainty quantification in time series prediction is challenging due to the temporal dependence and distribution shift on sequential data.<n>We propose itError-quantified Conformal Inference (ECI) by smoothing the quantile loss function.<n>ECI can achieve valid miscoverage control and output tighter prediction sets than other baselines.
  arXiv  Detail & Related papers  (2025-02-02T15:02:36Z)
• Conditional Testing based on Localized Conformal p-values [5.6779147365057305]
  We define the localized conformal p-values by inverting prediction intervals and prove their theoretical properties. These defined p-values are then applied to several conditional testing problems to illustrate their practicality.
  arXiv  Detail & Related papers  (2024-09-25T11:30:14Z)
• Uncertainty-Calibrated Test-Time Model Adaptation without Forgetting [65.21599711087538]
  Test-time adaptation (TTA) seeks to tackle potential distribution shifts between training and test data by adapting a given model w.r.t. any test sample.<n>Prior methods perform backpropagation for each test sample, resulting in unbearable optimization costs to many applications.<n>We propose an Efficient Anti-Forgetting Test-Time Adaptation (EATA) method which develops an active sample selection criterion to identify reliable and non-redundant samples.
  arXiv  Detail & Related papers  (2024-03-18T05:49:45Z)
• Generalized Robust Test-Time Adaptation in Continuous Dynamic Scenarios [18.527640606971563]
  Test-time adaptation (TTA) adapts pre-trained models to test distributions during the inference phase exclusively employing unlabeled test data streams. We propose a Generalized Robust Test-Time Adaptation (GRoTTA) method to effectively address the difficult problem.
  arXiv  Detail & Related papers  (2023-10-07T07:13:49Z)
• Sequential Kernelized Independence Testing [77.237958592189]
  We design sequential kernelized independence tests inspired by kernelized dependence measures.<n>We demonstrate the power of our approaches on both simulated and real data.
  arXiv  Detail & Related papers  (2022-12-14T18:08:42Z)
• Hierarchical Semi-Supervised Contrastive Learning for Contamination-Resistant Anomaly Detection [81.07346419422605]
  Anomaly detection aims at identifying deviant samples from the normal data distribution. Contrastive learning has provided a successful way to sample representation that enables effective discrimination on anomalies. We propose a novel hierarchical semi-supervised contrastive learning framework, for contamination-resistant anomaly detection.
  arXiv  Detail & Related papers  (2022-07-24T18:49:26Z)
• Approximate Conditional Coverage via Neural Model Approximations [0.030458514384586396]
  We analyze a data-driven procedure for obtaining empirically reliable approximate conditional coverage. We demonstrate the potential for substantial (and otherwise unknowable) under-coverage with split-conformal alternatives with marginal coverage guarantees.
  arXiv  Detail & Related papers  (2022-05-28T02:59:05Z)
• Training on Test Data with Bayesian Adaptation for Covariate Shift [96.3250517412545]
  Deep neural networks often make inaccurate predictions with unreliable uncertainty estimates. We derive a Bayesian model that provides for a well-defined relationship between unlabeled inputs under distributional shift and model parameters. We show that our method improves both accuracy and uncertainty estimation.
  arXiv  Detail & Related papers  (2021-09-27T01:09:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.