Beyond the Training Distribution: Evaluating Predictions Under Distribution Shift and Selection Bias

• URL: http://arxiv.org/abs/2606.14506v1
• Date: Fri, 12 Jun 2026 14:39:03 GMT
• Title: Beyond the Training Distribution: Evaluating Predictions Under Distribution Shift and Selection Bias
• Abstract summary: We study how a prediction model will perform in a new environment before deployment.<n>We present a double machine learning procedure for estimating the target risk of an arbitrary black-box prediction model.<n>We evaluate our estimator through experiments using the eICU electronic health records database.
• Score: 4.124127724629818
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Understanding how a prediction model will perform in a new environment before deployment is essential to preventing harm when algorithms inform decision-making. Two common sources of model performance degradation are (i) covariate shift, where the target covariate distribution differs from the source, and (ii) selective labels, where the observability of outcomes depends on historical decisions. We study pre-deployment model evaluation under the joint presence of covariate shift and labeling of outcomes selectively based on observed features. In particular, we present a double machine learning procedure for estimating the target risk of an arbitrary black-box prediction model under a general loss function. We show identification of this estimand under standard assumptions and derive a bias-corrected estimator based on the influence function of the target risk. Finally, we evaluate our estimator through experiments using the eICU electronic health records database, showing that it tracks the true target risk more accurately than methods that address either selective labels or covariate shift alone, as well as baselines that combine standard plug-in approaches.

Related papers

• Inference-Time Selective Debiasing to Enhance Fairness in Text Classification Models [27.578390085427156]
  We propose selective debiasing -- an inference-time safety mechanism designed to enhance the overall model quality.<n>We identify the potentially biased model predictions and, instead of discarding them, we remove bias from these predictions using LEACE -- a post-processing debiasing method.<n>Experiments on text classification datasets with encoder-based classification models demonstrate that selective debiasing helps to reduce the performance gap between post-processing methods and debiasing techniques.
  arXiv  Detail & Related papers  (2024-07-27T21:56:23Z)
• Conformalized Selective Regression [2.3964255330849356]
  We propose a novel approach to selective regression by leveraging conformal prediction. We show how our proposed approach, conformalized selective regression, demonstrates an advantage over multiple state-of-the-art baselines.
  arXiv  Detail & Related papers  (2024-02-26T04:43:50Z)
• Cal-SFDA: Source-Free Domain-adaptive Semantic Segmentation with Differentiable Expected Calibration Error [50.86671887712424]
  The prevalence of domain adaptive semantic segmentation has prompted concerns regarding source domain data leakage. To circumvent the requirement for source data, source-free domain adaptation has emerged as a viable solution. We propose a novel calibration-guided source-free domain adaptive semantic segmentation framework.
  arXiv  Detail & Related papers  (2023-08-06T03:28:34Z)
• Dirichlet-based Uncertainty Calibration for Active Domain Adaptation [33.33529827699169]
  Active domain adaptation (DA) aims to maximally boost the model adaptation on a new target domain by actively selecting limited target data to annotate. Traditional active learning methods may be less effective since they do not consider the domain shift issue. We propose a itDirichlet-based Uncertainty (DUC) approach for active DA, which simultaneously achieves the mitigation of miscalibration and the selection of informative target samples.
  arXiv  Detail & Related papers  (2023-02-27T14:33:29Z)
• Robust Design and Evaluation of Predictive Algorithms under Unobserved Confounding [6.925076885159747]
  We propose a framework for robust design and evaluation of predictive algorithms.<n>We show that varying confounding assumptions substantially affects credit risk predictions and fairness evaluations across income groups.
  arXiv  Detail & Related papers  (2022-12-19T20:41:44Z)
• Calibrated Selective Classification [34.08454890436067]
  We develop a new approach to selective classification in which we propose a method for rejecting examples with "uncertain" uncertainties. We present a framework for learning selectively calibrated models, where a separate selector network is trained to improve the selective calibration error of a given base model. We demonstrate the empirical effectiveness of our approach on multiple image classification and lung cancer risk assessment tasks.
  arXiv  Detail & Related papers  (2022-08-25T13:31:09Z)
• DEUP: Direct Epistemic Uncertainty Prediction [56.087230230128185]
  Epistemic uncertainty is part of out-of-sample prediction error due to the lack of knowledge of the learner. We propose a principled approach for directly estimating epistemic uncertainty by learning to predict generalization error and subtracting an estimate of aleatoric uncertainty.
  arXiv  Detail & Related papers  (2021-02-16T23:50:35Z)
• Selecting Treatment Effects Models for Domain Adaptation Using Causal Knowledge [82.5462771088607]
  We propose a novel model selection metric specifically designed for ITE methods under the unsupervised domain adaptation setting. In particular, we propose selecting models whose predictions of interventions' effects satisfy known causal structures in the target domain.
  arXiv  Detail & Related papers  (2021-02-11T21:03:14Z)
• Estimating Generalization under Distribution Shifts via Domain-Invariant Representations [75.74928159249225]
  We use a set of domain-invariant predictors as a proxy for the unknown, true target labels. The error of the resulting risk estimate depends on the target risk of the proxy model.
  arXiv  Detail & Related papers  (2020-07-06T17:21:24Z)
• Unlabelled Data Improves Bayesian Uncertainty Calibration under Covariate Shift [100.52588638477862]
  We develop an approximate Bayesian inference scheme based on posterior regularisation. We demonstrate the utility of our method in the context of transferring prognostic models of prostate cancer across globally diverse populations.
  arXiv  Detail & Related papers  (2020-06-26T13:50:19Z)
• Decision-Making with Auto-Encoding Variational Bayes [71.44735417472043]
  We show that a posterior approximation distinct from the variational distribution should be used for making decisions. Motivated by these theoretical results, we propose learning several approximate proposals for the best model. In addition to toy examples, we present a full-fledged case study of single-cell RNA sequencing.
  arXiv  Detail & Related papers  (2020-02-17T19:23:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.