SADA: Safe and Adaptive Inference with Multiple Black-Box Predictions

• URL: http://arxiv.org/abs/2509.21707v1
• Date: Fri, 26 Sep 2025 00:02:54 GMT
• Title: SADA: Safe and Adaptive Inference with Multiple Black-Box Predictions
• Authors: Jiawei Shan, Yiming Dong, Jiwei Zhao,
• Abstract summary: Real-world applications often face scarce labeled data due to the high cost and time requirements of gold-standard experiments.<n>We propose a novel approach that safely and adaptively aggregates multiple black-box predictions with unknown quality.
• Score: 4.100095195067256
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Real-world applications often face scarce labeled data due to the high cost and time requirements of gold-standard experiments, whereas unlabeled data are typically abundant. With the growing adoption of machine learning techniques, it has become increasingly feasible to generate multiple predicted labels using a variety of models and algorithms, including deep learning, large language models, and generative AI. In this paper, we propose a novel approach that safely and adaptively aggregates multiple black-box predictions with unknown quality while preserving valid statistical inference. Our method provides two key guarantees: (i) it never performs worse than using the labeled data alone, regardless of the quality of the predictions; and (ii) if any one of the predictions (without knowing which one) perfectly fits the ground truth, the algorithm adaptively exploits this to achieve either a faster convergence rate or the semiparametric efficiency bound. We demonstrate the effectiveness of the proposed algorithm through experiments on both synthetic and benchmark datasets.

Related papers

• Data-Efficient Prediction-Powered Calibration via Cross-Validation [35.04154147859041]
  This paper introduces a novel approach that efficiently utilizes limited calibration data to simultaneously fine-tune a predictor and estimate the bias of the synthetic labels.<n>The proposed method yields prediction sets with rigorous coverage guarantees for AI-generated decisions.
  arXiv  Detail & Related papers  (2025-07-27T13:31:02Z)
• Collaborative Prediction: To Join or To Disjoin Datasets [5.9697789282446605]
  We study the problem of developing practical algorithms that select appropriate dataset to minimize population loss.<n>By leveraging an oracle inequality and data-driven estimators, the algorithm reduces population loss with high probability.
  arXiv  Detail & Related papers  (2025-06-12T20:25:07Z)
• Conformal Validity Guarantees Exist for Any Data Distribution (and How to Find Them) [14.396431159723297]
  We show that conformal prediction can theoretically be extended to textitany joint data distribution. Although the most general case is exceedingly impractical to compute, for concrete practical applications we outline a procedure for deriving specific conformal algorithms.
  arXiv  Detail & Related papers  (2024-05-10T17:40:24Z)
• Overcoming Overconfidence for Active Learning [1.2776312584227847]
  We present two novel methods to address the problem of overconfidence that arises in the active learning scenario. The first is an augmentation strategy named Cross-Mix-and-Mix (CMaM), which aims to calibrate the model by expanding the limited training distribution. The second is a selection strategy named Ranked Margin Sampling (RankedMS), which prevents choosing data that leads to overly confident predictions.
  arXiv  Detail & Related papers  (2023-08-21T09:04:54Z)
• ASPEST: Bridging the Gap Between Active Learning and Selective Prediction [56.001808843574395]
  Selective prediction aims to learn a reliable model that abstains from making predictions when uncertain. Active learning aims to lower the overall labeling effort, and hence human dependence, by querying the most informative examples. In this work, we introduce a new learning paradigm, active selective prediction, which aims to query more informative samples from the shifted target domain.
  arXiv  Detail & Related papers  (2023-04-07T23:51:07Z)
• A Stable, Fast, and Fully Automatic Learning Algorithm for Predictive Coding Networks [65.34977803841007]
  Predictive coding networks are neuroscience-inspired models with roots in both Bayesian statistics and neuroscience. We show how by simply changing the temporal scheduling of the update rule for the synaptic weights leads to an algorithm that is much more efficient and stable than the original one.
  arXiv  Detail & Related papers  (2022-11-16T00:11:04Z)
• Efficient and Differentiable Conformal Prediction with General Function Classes [96.74055810115456]
  We propose a generalization of conformal prediction to multiple learnable parameters. We show that it achieves approximate valid population coverage and near-optimal efficiency within class. Experiments show that our algorithm is able to learn valid prediction sets and improve the efficiency significantly.
  arXiv  Detail & Related papers  (2022-02-22T18:37:23Z)
• Dash: Semi-Supervised Learning with Dynamic Thresholding [72.74339790209531]
  We propose a semi-supervised learning (SSL) approach that uses unlabeled examples to train models. Our proposed approach, Dash, enjoys its adaptivity in terms of unlabeled data selection.
  arXiv  Detail & Related papers  (2021-09-01T23:52:29Z)
• RATT: Leveraging Unlabeled Data to Guarantee Generalization [96.08979093738024]
  We introduce a method that leverages unlabeled data to produce generalization bounds. We prove that our bound is valid for 0-1 empirical risk minimization. This work provides practitioners with an option for certifying the generalization of deep nets even when unseen labeled data is unavailable.
  arXiv  Detail & Related papers  (2021-05-01T17:05:29Z)
• Ambiguity in Sequential Data: Predicting Uncertain Futures with Recurrent Models [110.82452096672182]
  We propose an extension of the Multiple Hypothesis Prediction (MHP) model to handle ambiguous predictions with sequential data. We also introduce a novel metric for ambiguous problems, which is better suited to account for uncertainties.
  arXiv  Detail & Related papers  (2020-03-10T09:15:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.