Anytime-Valid Confirmation of Label-Shift Corrections

• URL: http://arxiv.org/abs/2606.14028v1
• Date: Fri, 12 Jun 2026 02:03:21 GMT
• Title: Anytime-Valid Confirmation of Label-Shift Corrections
• Abstract summary: In small-batch scientific deployments, labeled target outcomes may be too scarce for reliable shift estimation.<n>We show that the per-observation likelihood ratio between a label-shift-corrected predictive and the source predictive is a conditional e-value.
• Score: 0.6599344783327052
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In small-batch scientific deployments, labeled target outcomes may be too scarce for reliable shift estimation even when unlabeled target inputs are available. We address the complementary setting where the practitioner has a pre-specified label-shift correction from domain knowledge and asks whether incoming labeled outcomes support it. We show that the per-observation likelihood ratio between a label-shift-corrected predictive and the source predictive is a conditional e-value, so its running product is a nonnegative martingale and Ville's inequality yields an anytime-valid confirmation rule. The log martingale equals the cumulative negative log-predictive density (NLPD) gap between the source and the corrected predictive, converting routine model monitoring into a formal sequential test. Rejection means the incoming data support the posited correction relative to the source predictive, but it is not a precise estimate of the degree of shift. Closed forms are available for GP sources with Gaussian label-shift ratios. GP regression simulations validate Type I control, finite-sample power, miscalibration sensitivity, and the small-batch advantage of a reliable prior over label-based re-estimation.

Related papers

• Split Conformal Prediction with Label-Shift-Adjusted Bayesian Scores [2.2400646874013037]
  Under label shift, standard conformal procedures no longer maintain their intended coverage behavior.<n>We propose the emphLabel-Shift-Adjusted Bayesian Score (LSA score), a nonconformity score derived from a posterior predictive identity.<n>LSA score consistently yields shorter intervals than residual-based and source-based Bayesian scores.
  arXiv  Detail & Related papers  (2026-09-11T03:06:31Z)
• Anytime-Valid Confirmation of Covariate Balance for Prespecified Corrections [0.6599344783327052]
  We study anytime-valid confirmation of prespecified corrections from sequential unlabeled target inputs.<n>Experiments illustrate false-confirmation control, locality to the balancing-function class, KL-drift diagnostics, acceptable-region monitoring, finite-source effects.
  arXiv  Detail & Related papers  (2026-07-25T11:21:17Z)
• Prediction-Powered Conditional Inference [15.801272841475681]
  We study prediction-powered conditional inference in the setting where labeled data are scarce.<n>Our approach combines localization with prediction-based variance reduction.<n> Numerical experiments on simulated and real datasets demonstrate valid conditional coverage.
  arXiv  Detail & Related papers  (2026-03-05T17:13:06Z)
• Coverage Guarantees for Pseudo-Calibrated Conformal Prediction under Distribution Shift [1.5861469511290378]
  Conformal prediction offers marginal coverage guarantees if the data distribution shifts.<n>We analyze the use of pseudo-calibration as a tool to counter this performance loss.<n>We propose a source-tuned pseudo-calibration algorithm that interpolates between hard pseudo-labels and randomized labels.
  arXiv  Detail & Related papers  (2026-02-16T16:48:39Z)
• Non-exchangeable Conformal Prediction with Optimal Transport: Tackling Distribution Shifts with Unlabeled Data [13.788274786492286]
  Conformal prediction is a distribution-free uncertainty quantification method that has gained popularity in the machine learning community.<n>Its most common variant, dubbed split conformal prediction, is also computationally efficient as it boils down to collecting statistics of the model predictions on some calibration data not yet seen by the model.<n>We show that it is possible to estimate the loss in coverage and mitigate arbitrary distribution shifts, offering a principled and broadly applicable solution.
  arXiv  Detail & Related papers  (2025-07-14T16:10:55Z)
• Provably Reliable Conformal Prediction Sets in the Presence of Data Poisoning [53.42244686183879]
  Conformal prediction provides model-agnostic and distribution-free uncertainty quantification.<n>Yet, conformal prediction is not reliable under poisoning attacks where adversaries manipulate both training and calibration data.<n>We propose reliable prediction sets (RPS): the first efficient method for constructing conformal prediction sets with provable reliability guarantees under poisoning.
  arXiv  Detail & Related papers  (2024-10-13T15:37:11Z)
• Predicting generalization performance with correctness discriminators [56.948321040125734]
  We present a novel model that establishes upper and lower bounds on the accuracy, without requiring gold labels for the unseen data.<n>We show across a variety of tagging, parsing, and semantic parsing tasks that the gold accuracy is reliably between the predicted upper and lower bounds.
  arXiv  Detail & Related papers  (2023-11-15T22:43:42Z)
• Leveraging Unlabeled Data to Predict Out-of-Distribution Performance [63.740181251997306]
  Real-world machine learning deployments are characterized by mismatches between the source (training) and target (test) distributions. In this work, we investigate methods for predicting the target domain accuracy using only labeled source data and unlabeled target data. We propose Average Thresholded Confidence (ATC), a practical method that learns a threshold on the model's confidence, predicting accuracy as the fraction of unlabeled examples.
  arXiv  Detail & Related papers  (2022-01-11T23:01:12Z)
• Training on Test Data with Bayesian Adaptation for Covariate Shift [96.3250517412545]
  Deep neural networks often make inaccurate predictions with unreliable uncertainty estimates. We derive a Bayesian model that provides for a well-defined relationship between unlabeled inputs under distributional shift and model parameters. We show that our method improves both accuracy and uncertainty estimation.
  arXiv  Detail & Related papers  (2021-09-27T01:09:08Z)
• Tail-to-Tail Non-Autoregressive Sequence Prediction for Chinese Grammatical Error Correction [49.25830718574892]
  We present a new framework named Tail-to-Tail (textbfTtT) non-autoregressive sequence prediction. Considering that most tokens are correct and can be conveyed directly from source to target, and the error positions can be estimated and corrected. Experimental results on standard datasets, especially on the variable-length datasets, demonstrate the effectiveness of TtT in terms of sentence-level Accuracy, Precision, Recall, and F1-Measure.
  arXiv  Detail & Related papers  (2021-06-03T05:56:57Z)
• Distribution-free uncertainty quantification for classification under label shift [105.27463615756733]
  We focus on uncertainty quantification (UQ) for classification problems via two avenues. We first argue that label shift hurts UQ, by showing degradation in coverage and calibration. We examine these techniques theoretically in a distribution-free framework and demonstrate their excellent practical performance.
  arXiv  Detail & Related papers  (2021-03-04T20:51:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.