Split Conformal Prediction with Label-Shift-Adjusted Bayesian Scores

• URL: http://arxiv.org/abs/2609.12386v1
• Date: Fri, 11 Sep 2026 03:06:31 GMT
• Title: Split Conformal Prediction with Label-Shift-Adjusted Bayesian Scores
• Abstract summary: Under label shift, standard conformal procedures no longer maintain their intended coverage behavior.<n>We propose the emphLabel-Shift-Adjusted Bayesian Score (LSA score), a nonconformity score derived from a posterior predictive identity.<n>LSA score consistently yields shorter intervals than residual-based and source-based Bayesian scores.
• Score: 2.2400646874013037
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Conformal prediction provides distribution-free uncertainty quantification under exchangeability. However, this assumption is violated by label shift, where the marginal distribution of labels changes while the conditional distribution of inputs given labels remains stable. Under such shifts, standard conformal procedures no longer maintain their intended coverage behavior. Existing approaches address this via importance weighting. They pair the reweighting with residual-based nonconformity scores that ignore predictive uncertainty. The resulting intervals have uniform width. Bayesian conformal methods produce adaptive intervals by leveraging predictive distributions. They evaluate conformity under the source predictive, which is misaligned with the target domain under label shift. We propose the \emph{Label-Shift-Adjusted Bayesian Score} (LSA score), a nonconformity score derived from a posterior predictive tilting identity. This identity shows that the target predictive is an importance-weighted transformation of the source predictive. We use it to derive a direct correction to the Bayesian score. We evaluate the method on molecular property prediction under controlled label shift. The LSA score consistently yields shorter intervals than residual-based and source-based Bayesian scores. Coverage in the target domain remains comparable. Under stronger shift, all methods incur some coverage loss due to pseudo-label-based density-ratio estimation. The LSA score is defined for any source predictive with a tractable log-density. We instantiate it with Bayesian Ridge Regression, where the correction admits a closed form.

Related papers

• Anytime-Valid Confirmation of Label-Shift Corrections [0.6599344783327052]
  In small-batch scientific deployments, labeled target outcomes may be too scarce for reliable shift estimation.<n>We show that the per-observation likelihood ratio between a label-shift-corrected predictive and the source predictive is a conditional e-value.
  arXiv  Detail & Related papers  (2026-06-12T02:03:21Z)
• Empirical Bayes Conformal Prediction for Vision and Language Models [20.40115871314124]
  Conformal prediction (CP) gives distribution-free coverage for modern vision and language models.<n>Standard CP uses one realization, while average-then-calibrate variants smooth multiple realizations into a point estimate.<n>We describe an empirical Bayes conformal prediction framework that uses $r$-values to convert score variability into an uncertainty informed nonconformity score.
  arXiv  Detail & Related papers  (2026-05-22T03:17:14Z)
• Coverage Guarantees for Pseudo-Calibrated Conformal Prediction under Distribution Shift [1.5861469511290378]
  Conformal prediction offers marginal coverage guarantees if the data distribution shifts.<n>We analyze the use of pseudo-calibration as a tool to counter this performance loss.<n>We propose a source-tuned pseudo-calibration algorithm that interpolates between hard pseudo-labels and randomized labels.
  arXiv  Detail & Related papers  (2026-02-16T16:48:39Z)
• Conformal Prediction Sets with Improved Conditional Coverage using Trust Scores [52.92618442300405]
  It is impossible to achieve exact, distribution-free conditional coverage in finite samples.<n>We propose an alternative conformal prediction algorithm that targets coverage where it matters most.
  arXiv  Detail & Related papers  (2025-01-17T12:01:56Z)
• Contrastive Conditional Alignment based on Label Shift Calibration for Imbalanced Domain Adaptation [16.944918133828722]
  We propose contrastive conditional alignment based on label shift calibration (CCA-LSC) for unsupervised domain adaptation (UDA)<n>Our method outperforms existing UDA and IDA methods on benchmarks with both label shift and covariate shift.
  arXiv  Detail & Related papers  (2024-12-29T03:34:31Z)
• Equal Opportunity of Coverage in Fair Regression [50.76908018786335]
  We study fair machine learning (ML) under predictive uncertainty to enable reliable and trustworthy decision-making. We propose Equal Opportunity of Coverage (EOC) that aims to achieve two properties: (1) coverage rates for different groups with similar outcomes are close, and (2) the coverage rate for the entire population remains at a predetermined level.
  arXiv  Detail & Related papers  (2023-11-03T21:19:59Z)
• Generating Unbiased Pseudo-labels via a Theoretically Guaranteed Chebyshev Constraint to Unify Semi-supervised Classification and Regression [57.17120203327993]
  threshold-to-pseudo label process (T2L) in classification uses confidence to determine the quality of label. In nature, regression also requires unbiased methods to generate high-quality labels. We propose a theoretically guaranteed constraint for generating unbiased labels based on Chebyshev's inequality.
  arXiv  Detail & Related papers  (2023-11-03T08:39:35Z)
• PAC Prediction Sets Under Label Shift [52.30074177997787]
  Prediction sets capture uncertainty by predicting sets of labels rather than individual labels. We propose a novel algorithm for constructing prediction sets with PAC guarantees in the label shift setting. We evaluate our approach on five datasets.
  arXiv  Detail & Related papers  (2023-10-19T17:57:57Z)
• Distribution-free uncertainty quantification for classification under label shift [105.27463615756733]
  We focus on uncertainty quantification (UQ) for classification problems via two avenues. We first argue that label shift hurts UQ, by showing degradation in coverage and calibration. We examine these techniques theoretically in a distribution-free framework and demonstrate their excellent practical performance.
  arXiv  Detail & Related papers  (2021-03-04T20:51:03Z)
• Exploiting Sample Uncertainty for Domain Adaptive Person Re-Identification [137.9939571408506]
  We estimate and exploit the credibility of the assigned pseudo-label of each sample to alleviate the influence of noisy labels. Our uncertainty-guided optimization brings significant improvement and achieves the state-of-the-art performance on benchmark datasets.
  arXiv  Detail & Related papers  (2020-12-16T04:09:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.