Inferential Evaluation of Surrogate-Derived Models under Covariate Shift

• URL: http://arxiv.org/abs/2608.15783v1
• Date: Sun, 16 Aug 2026 15:05:38 GMT
• Title: Inferential Evaluation of Surrogate-Derived Models under Covariate Shift
• Abstract summary: In transfer-learning settings, a model derived from surrogate labels may be deployed in a target population where gold-standard outcomes are unobserved.<n>We study three-sample setting with a small gold-labeled source, a larger surrogate-labeled source, and an unlabeled target.<n>Under conditional transportability, we evaluate the surrogate-derived model against the latent gold-standard outcome in the target population.
• Score: 5.27896359595804
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In transfer-learning settings, a model derived from abundant surrogate labels may be deployed in a target population where gold-standard outcomes are unobserved. Evaluating its target performance is essential for determining whether decisions based on the model remain reliable, yet it is difficult when gold labels are scarce, and covariate distributions differ across data sources. We study a three-sample setting with a small gold-labeled source, a larger surrogate-labeled source, and an unlabeled target. Under conditional transportability, we evaluate the surrogate-derived model against the latent gold-standard outcome in the target population. We propose cross-fitted estimators that transport information from the two labeled sources through source-specific density ratios. We also combine outcome-regression augmentation with a kernel correction for estimating the model near a threshold, accounting for uncertainty from all three samples. We establish asymptotically linear inference for TPR and FPR, consistency and pointwise inference for the ROC curve, and asymptotically normal inference for AUC. Simulations assess bias, coverage, and sensitivity to bandwidth and relative sample sizes. A retrospective temporal validation on Chatbot Arena and a semi-synthetic ACS-Income study provide validation in real-world AI applications.

Related papers

• Uncertainty-based Debiasing and Unlearning for Decontamination [12.593872705071114]
  Uncertainty-Based Decontamination (UBD) estimates per-sample memorization without requiring a contaminated model or knowledge of which samples are contaminated.<n>UBD produces per-sample output distributions substantially closer to those of an uncontaminated model than paraphrasing or choice-permutation baselines.
  arXiv  Detail & Related papers  (2026-06-22T13:26:29Z)
• Beyond the Training Distribution: Evaluating Predictions Under Distribution Shift and Selection Bias [4.124127724629818]
  We study how a prediction model will perform in a new environment before deployment.<n>We present a double machine learning procedure for estimating the target risk of an arbitrary black-box prediction model.<n>We evaluate our estimator through experiments using the eICU electronic health records database.
  arXiv  Detail & Related papers  (2026-06-12T14:39:03Z)
• Pseudo-Labeling for Unsupervised Domain Adaptation with Kernel GLMs [4.153834414190214]
  We propose a principled framework for unsupervised domain adaptation.<n>Our goal is to minimize prediction error in the target domain by leveraging labeled source data and unlabeled target data.<n>Experiments on synthetic and real datasets demonstrate consistent performance gains over source-only baselines.
  arXiv  Detail & Related papers  (2026-03-19T19:31:47Z)
• Heterogeneous Multisource Transfer Learning via Model Averaging for Positive-Unlabeled Data [2.030810815519794]
  We propose a novel transfer learning framework that integrates information from heterogeneous data sources without direct data sharing.<n>For each source domain type, a tailored logistic regression model is conducted, and knowledge is transferred to the PU target domain through model averaging.<n>Our method outperforms other comparative methods in terms of predictive accuracy and robustness, especially under limited labeled data and heterogeneous environments.
  arXiv  Detail & Related papers  (2025-11-14T03:15:31Z)
• Theory on Score-Mismatched Diffusion Models and Zero-Shot Conditional Samplers [49.97755400231656]
  We present the first performance guarantee with explicit dimensional dependencies for general score-mismatched diffusion samplers.<n>We show that score mismatches result in an distributional bias between the target and sampling distributions, proportional to the accumulated mismatch between the target and training distributions.<n>This result can be directly applied to zero-shot conditional samplers for any conditional model, irrespective of measurement noise.
  arXiv  Detail & Related papers  (2024-10-17T16:42:12Z)
• The Decaying Missing-at-Random Framework: Model Doubly Robust Causal Inference with Partially Labeled Data [8.916614661563893]
  We introduce a missing-at-random (decaying MAR) framework and associated approaches for doubly robust causal inference.<n>This simultaneously addresses selection bias in the labeling mechanism and the extreme imbalance between labeled and unlabeled groups.<n>To ensure robust causal conclusions, we propose a bias-reduced SS estimator for the average treatment effect.
  arXiv  Detail & Related papers  (2023-05-22T07:37:12Z)
• Breaking the Spurious Causality of Conditional Generation via Fairness Intervention with Corrective Sampling [77.15766509677348]
  Conditional generative models often inherit spurious correlations from the training dataset. This can result in label-conditional distributions that are imbalanced with respect to another latent attribute. We propose a general two-step strategy to mitigate this issue.
  arXiv  Detail & Related papers  (2022-12-05T08:09:33Z)
• Leveraging Unlabeled Data to Predict Out-of-Distribution Performance [63.740181251997306]
  Real-world machine learning deployments are characterized by mismatches between the source (training) and target (test) distributions. In this work, we investigate methods for predicting the target domain accuracy using only labeled source data and unlabeled target data. We propose Average Thresholded Confidence (ATC), a practical method that learns a threshold on the model's confidence, predicting accuracy as the fraction of unlabeled examples.
  arXiv  Detail & Related papers  (2022-01-11T23:01:12Z)
• How Faithful is your Synthetic Data? Sample-level Metrics for Evaluating and Auditing Generative Models [95.8037674226622]
  We introduce a 3-dimensional evaluation metric that characterizes the fidelity, diversity and generalization performance of any generative model in a domain-agnostic fashion. Our metric unifies statistical divergence measures with precision-recall analysis, enabling sample- and distribution-level diagnoses of model fidelity and diversity.
  arXiv  Detail & Related papers  (2021-02-17T18:25:30Z)
• Exploiting Sample Uncertainty for Domain Adaptive Person Re-Identification [137.9939571408506]
  We estimate and exploit the credibility of the assigned pseudo-label of each sample to alleviate the influence of noisy labels. Our uncertainty-guided optimization brings significant improvement and achieves the state-of-the-art performance on benchmark datasets.
  arXiv  Detail & Related papers  (2020-12-16T04:09:04Z)
• Unlabelled Data Improves Bayesian Uncertainty Calibration under Covariate Shift [100.52588638477862]
  We develop an approximate Bayesian inference scheme based on posterior regularisation. We demonstrate the utility of our method in the context of transferring prognostic models of prostate cancer across globally diverse populations.
  arXiv  Detail & Related papers  (2020-06-26T13:50:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.