Semi-Supervised Risk Control via Prediction-Powered Inference

• URL: http://arxiv.org/abs/2412.11174v1
• Date: Sun, 15 Dec 2024 13:00:23 GMT
• Title: Semi-Supervised Risk Control via Prediction-Powered Inference
• Authors: Bat-Sheva Einbinder, Liran Ringel, Yaniv Romano,
• Abstract summary: Risk-controlling prediction sets (RCPS) is a tool for transforming the output of any machine learning model to design a predictive rule with rigorous error rate control. We introduce a semi-supervised calibration procedure that leverages unlabeled data to rigorously tune the hyper- parameter. Our procedure builds upon the prediction-powered inference framework, carefully tailoring it to risk-controlling tasks.
• Score: 14.890609936348277
• License:
• Abstract: The risk-controlling prediction sets (RCPS) framework is a general tool for transforming the output of any machine learning model to design a predictive rule with rigorous error rate control. The key idea behind this framework is to use labeled hold-out calibration data to tune a hyper-parameter that affects the error rate of the resulting prediction rule. However, the limitation of such a calibration scheme is that with limited hold-out data, the tuned hyper-parameter becomes noisy and leads to a prediction rule with an error rate that is often unnecessarily conservative. To overcome this sample-size barrier, we introduce a semi-supervised calibration procedure that leverages unlabeled data to rigorously tune the hyper-parameter without compromising statistical validity. Our procedure builds upon the prediction-powered inference framework, carefully tailoring it to risk-controlling tasks. We demonstrate the benefits and validity of our proposal through two real-data experiments: few-shot image classification and early time series classification.

Related papers

• Robust Conformal Outlier Detection under Contaminated Reference Data [20.864605211132663]
  Conformal prediction is a flexible framework for calibrating machine learning predictions. In outlier detection, this calibration relies on a reference set of labeled inlier data to control the type-I error rate. This paper analyzes the impact of contamination on the validity of conformal methods.
  arXiv  Detail & Related papers  (2025-02-07T10:23:25Z)
• Parametric Scaling Law of Tuning Bias in Conformal Prediction [11.970092440023956]
  We find that the tuning bias - the coverage gap introduced by leveraging the same dataset for tuning and calibration - is negligible for simple parameter tuning in many conformal prediction methods. We establish a theoretical framework to quantify the tuning bias and provide rigorous proof for the scaling law of the tuning bias by deriving its upper bound.
  arXiv  Detail & Related papers  (2025-02-05T09:26:47Z)
• Conformal Generative Modeling with Improved Sample Efficiency through Sequential Greedy Filtering [55.15192437680943]
  Generative models lack rigorous statistical guarantees for their outputs. We propose a sequential conformal prediction method producing prediction sets that satisfy a rigorous statistical guarantee. This guarantee states that with high probability, the prediction sets contain at least one admissible (or valid) example.
  arXiv  Detail & Related papers  (2024-10-02T15:26:52Z)
• Towards Certification of Uncertainty Calibration under Adversarial Attacks [96.48317453951418]
  We show that attacks can significantly harm calibration, and thus propose certified calibration as worst-case bounds on calibration under adversarial perturbations. We propose novel calibration attacks and demonstrate how they can improve model calibration through textitadversarial calibration training
  arXiv  Detail & Related papers  (2024-05-22T18:52:09Z)
• U-Calibration: Forecasting for an Unknown Agent [29.3181385170725]
  We show that optimizing forecasts for a single scoring rule cannot guarantee low regret for all possible agents. We present a new metric for evaluating forecasts that we call U-calibration, equal to the maximal regret of the sequence of forecasts.
  arXiv  Detail & Related papers  (2023-06-30T23:05:26Z)
• Improving Adaptive Conformal Prediction Using Self-Supervised Learning [72.2614468437919]
  We train an auxiliary model with a self-supervised pretext task on top of an existing predictive model and use the self-supervised error as an additional feature to estimate nonconformity scores. We empirically demonstrate the benefit of the additional information using both synthetic and real data on the efficiency (width), deficit, and excess of conformal prediction intervals.
  arXiv  Detail & Related papers  (2023-02-23T18:57:14Z)
• Loss-Controlling Calibration for Predictive Models [5.51361762392299]
  We propose a learning framework for calibrating predictive models to make loss-controlling prediction for exchangeable data. By comparison, the predictors built by the proposed loss-controlling approach are not limited to set predictors. Our proposed method is applied to selective regression and high-impact weather forecasting problems.
  arXiv  Detail & Related papers  (2023-01-11T09:44:55Z)
• Few-Shot Calibration of Set Predictors via Meta-Learned Cross-Validation-Based Conformal Prediction [33.33774397643919]
  This paper introduces a novel meta-learning solution that aims at reducing the set prediction size. It builds on cross-validation-based CP, rather than the less efficient validation-based CP. It preserves formal per-task calibration guarantees, rather than less stringent task-marginal guarantees.
  arXiv  Detail & Related papers  (2022-10-06T17:21:03Z)
• Distribution-Free, Risk-Controlling Prediction Sets [112.9186453405701]
  We show how to generate set-valued predictions from a black-box predictor that control the expected loss on future test points at a user-specified level. Our approach provides explicit finite-sample guarantees for any dataset by using a holdout set to calibrate the size of the prediction sets.
  arXiv  Detail & Related papers  (2021-01-07T18:59:33Z)
• AutoCP: Automated Pipelines for Accurate Prediction Intervals [84.16181066107984]
  This paper proposes an AutoML framework called Automatic Machine Learning for Conformal Prediction (AutoCP) Unlike the familiar AutoML frameworks that attempt to select the best prediction model, AutoCP constructs prediction intervals that achieve the user-specified target coverage rate. We tested AutoCP on a variety of datasets and found that it significantly outperforms benchmark algorithms.
  arXiv  Detail & Related papers  (2020-06-24T23:13:11Z)
• Understanding and Mitigating the Tradeoff Between Robustness and Accuracy [88.51943635427709]
  Adversarial training augments the training set with perturbations to improve the robust error. We show that the standard error could increase even when the augmented perturbations have noiseless observations from the optimal linear predictor.
  arXiv  Detail & Related papers  (2020-02-25T08:03:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.