Data-Efficient Prediction-Powered Calibration via Cross-Validation

• URL: http://arxiv.org/abs/2507.20268v1
• Date: Sun, 27 Jul 2025 13:31:02 GMT
• Title: Data-Efficient Prediction-Powered Calibration via Cross-Validation
• Authors: Seonghoon Yoo, Houssem Sifaou, Sangwoo Park, Joonhyuk Kang, Osvaldo Simeone,
• Abstract summary: This paper introduces a novel approach that efficiently utilizes limited calibration data to simultaneously fine-tune a predictor and estimate the bias of the synthetic labels.<n>The proposed method yields prediction sets with rigorous coverage guarantees for AI-generated decisions.
• Score: 35.04154147859041
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Calibration data are necessary to formally quantify the uncertainty of the decisions produced by an existing artificial intelligence (AI) model. To overcome the common issue of scarce calibration data, a promising approach is to employ synthetic labels produced by a (generally different) predictive model. However, fine-tuning the label-generating predictor on the inference task of interest, as well as estimating the residual bias of the synthetic labels, demand additional data, potentially exacerbating the calibration data scarcity problem. This paper introduces a novel approach that efficiently utilizes limited calibration data to simultaneously fine-tune a predictor and estimate the bias of the synthetic labels. The proposed method yields prediction sets with rigorous coverage guarantees for AI-generated decisions. Experimental results on an indoor localization problem validate the effectiveness and performance gains of our solution.

Related papers

• Anytime-valid, Bayes-assisted,Prediction-Powered Inference [0.0]
  Given a large pool of unlabelled data, prediction-powered inference (PPI) leverages machine learning predictions to increase statistical efficiency.<n>We extend the PPI framework to the sequential setting, where labelled and unlabelled datasets grow over time.<n>We propose prediction-powered confidence sequence procedures that are valid uniformly over time and naturally accommodate prior knowledge on the quality of the predictions.
  arXiv  Detail & Related papers  (2025-05-23T15:05:49Z)
• Beyond One-Hot Labels: Semantic Mixing for Model Calibration [22.39558434131574]
  We present textbfCalibration-aware Semantic Mixing (CSM), a novel framework that generates training samples with mixed class characteristics.<n>We show that CSM achieves superior calibration compared to the state-of-the-art calibration approaches.
  arXiv  Detail & Related papers  (2025-04-18T08:26:18Z)
• Noise-Adaptive Conformal Classification with Marginal Coverage [53.74125453366155]
  We introduce an adaptive conformal inference method capable of efficiently handling deviations from exchangeability caused by random label noise.<n>We validate our method through extensive numerical experiments demonstrating its effectiveness on synthetic and real data sets.
  arXiv  Detail & Related papers  (2025-01-29T23:55:23Z)
• A Conformal Approach to Feature-based Newsvendor under Model Misspecification [2.801095519296785]
  We propose a model-free and distribution-free framework inspired by conformal prediction.<n>We validate our framework using both simulated data and a real-world dataset from the Capital Bikeshare program in Washington, D.C.
  arXiv  Detail & Related papers  (2024-12-17T18:34:43Z)
• Fill In The Gaps: Model Calibration and Generalization with Synthetic Data [2.89287673224661]
  We propose a calibration method that incorporates synthetic data without compromising accuracy. We derive the expected calibration error (ECE) bound using the Probably Approximately Correct (PAC) learning framework. We observed an average up to 34% increase in accuracy and 33% decrease in ECE.
  arXiv  Detail & Related papers  (2024-10-07T23:06:42Z)
• Learning with Imbalanced Noisy Data by Preventing Bias in Sample Selection [82.43311784594384]
  Real-world datasets contain not only noisy labels but also class imbalance. We propose a simple yet effective method to address noisy labels in imbalanced datasets.
  arXiv  Detail & Related papers  (2024-02-17T10:34:53Z)
• Improving Adaptive Conformal Prediction Using Self-Supervised Learning [72.2614468437919]
  We train an auxiliary model with a self-supervised pretext task on top of an existing predictive model and use the self-supervised error as an additional feature to estimate nonconformity scores. We empirically demonstrate the benefit of the additional information using both synthetic and real data on the efficiency (width), deficit, and excess of conformal prediction intervals.
  arXiv  Detail & Related papers  (2023-02-23T18:57:14Z)
• Conformal Prediction Under Feedback Covariate Shift for Biomolecular Design [56.86533144730384]
  We introduce a method to quantify predictive uncertainty in settings where the training and test data are statistically dependent.<n>As a motivating use case, we demonstrate with several real data sets how our method quantifies uncertainty for the predicted fitness of designed proteins.
  arXiv  Detail & Related papers  (2022-02-08T02:59:12Z)
• Theoretical characterization of uncertainty in high-dimensional linear classification [24.073221004661427]
  We show that uncertainty for learning from limited number of samples of high-dimensional input data and labels can be obtained by the approximate message passing algorithm. We discuss how over-confidence can be mitigated by appropriately regularising, and show that cross-validating with respect to the loss leads to better calibration than with the 0/1 error.
  arXiv  Detail & Related papers  (2022-02-07T15:32:07Z)
• Unsupervised Calibration under Covariate Shift [92.02278658443166]
  We introduce the problem of calibration under domain shift and propose an importance sampling based approach to address it. We evaluate and discuss the efficacy of our method on both real-world datasets and synthetic datasets.
  arXiv  Detail & Related papers  (2020-06-29T21:50:07Z)
• Unlabelled Data Improves Bayesian Uncertainty Calibration under Covariate Shift [100.52588638477862]
  We develop an approximate Bayesian inference scheme based on posterior regularisation. We demonstrate the utility of our method in the context of transferring prognostic models of prostate cancer across globally diverse populations.
  arXiv  Detail & Related papers  (2020-06-26T13:50:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.