Cross-Validation Conformal Risk Control
- URL: http://arxiv.org/abs/2401.11974v2
- Date: Wed, 1 May 2024 15:33:36 GMT
- Title: Cross-Validation Conformal Risk Control
- Authors: Kfir M. Cohen, Sangwoo Park, Osvaldo Simeone, Shlomo Shamai,
- Abstract summary: Conformal risk control (CRC) is a recently proposed technique that applies post-hoc to a conventional point predictor to provide calibration guarantees.
In this paper, a novel CRC method is introduced that is based on cross-validation, rather than on validation as the original CRC.
CV-CRC is proved to offer theoretical guarantees on the average risk of the set predictor.
- Score: 40.2365781482563
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Conformal risk control (CRC) is a recently proposed technique that applies post-hoc to a conventional point predictor to provide calibration guarantees. Generalizing conformal prediction (CP), with CRC, calibration is ensured for a set predictor that is extracted from the point predictor to control a risk function such as the probability of miscoverage or the false negative rate. The original CRC requires the available data set to be split between training and validation data sets. This can be problematic when data availability is limited, resulting in inefficient set predictors. In this paper, a novel CRC method is introduced that is based on cross-validation, rather than on validation as the original CRC. The proposed cross-validation CRC (CV-CRC) extends a version of the jackknife-minmax from CP to CRC, allowing for the control of a broader range of risk functions. CV-CRC is proved to offer theoretical guarantees on the average risk of the set predictor. Furthermore, numerical experiments show that CV-CRC can reduce the average set size with respect to CRC when the available data are limited.
Related papers
- Conformal Distributed Remote Inference in Sensor Networks Under Reliability and Communication Constraints [61.62410595953275]
Communication-constrained distributed conformal risk control (CD-CRC)
CD-CRC is a novel decision-making framework for sensor networks under communication constraints.
arXiv Detail & Related papers (2024-09-12T10:12:43Z) - Risk and cross validation in ridge regression with correlated samples [72.59731158970894]
We provide training examples for the in- and out-of-sample risks of ridge regression when the data points have arbitrary correlations.
We further extend our analysis to the case where the test point has non-trivial correlations with the training set, setting often encountered in time series forecasting.
We validate our theory across a variety of high dimensional data.
arXiv Detail & Related papers (2024-08-08T17:27:29Z) - Localized Adaptive Risk Control [35.88667386998423]
Localized Adaptive Risk Control (L-ARC) is an online calibration scheme that targets statistical localized risk guarantees.
L-ARC is demonstrated via experiments to produce prediction sets with risk guarantees across different data subpopulations.
arXiv Detail & Related papers (2024-05-13T17:48:45Z) - CAP: A General Algorithm for Online Selective Conformal Prediction with FCR Control [4.137346786534721]
It is important to control the real-time false coverage-statement rate (FCR) which measures the overall miscoverage level.
We develop a general framework named CAP that performs an adaptive pick rule on historical data to construct a calibration set.
We prove that CAP can achieve an exact selection-conditional coverage guarantee in the finite-sample and distribution-free regimes.
arXiv Detail & Related papers (2024-03-12T15:07:20Z) - Forking Uncertainties: Reliable Prediction and Model Predictive Control
with Sequence Models via Conformal Risk Control [40.918012779935246]
We introduce a novel post-hoc calibration procedure that operates on the predictions produced by any pre-designed probabilistic forecaster to yield reliable error bars.
Unlike the state of the art, PTS-CRC can satisfy reliability definitions beyond coverage.
We experimentally validate the performance of PTS-CRC prediction and control by studying a number of use cases in the context of wireless networking.
arXiv Detail & Related papers (2023-10-16T11:35:41Z) - Safe Deployment for Counterfactual Learning to Rank with Exposure-Based
Risk Minimization [63.93275508300137]
We introduce a novel risk-aware Counterfactual Learning To Rank method with theoretical guarantees for safe deployment.
Our experimental results demonstrate the efficacy of our proposed method, which is effective at avoiding initial periods of bad performance when little data is available.
arXiv Detail & Related papers (2023-04-26T15:54:23Z) - Bayes risk CTC: Controllable CTC alignment in Sequence-to-Sequence tasks [63.189632935619535]
Bayes risk CTC (BRCTC) is proposed to enforce the desired characteristics of the predicted alignment.
By using BRCTC with another preference for early emissions, we obtain an improved performance-latency trade-off for online models.
arXiv Detail & Related papers (2022-10-14T03:55:36Z) - Few-Shot Calibration of Set Predictors via Meta-Learned
Cross-Validation-Based Conformal Prediction [33.33774397643919]
This paper introduces a novel meta-learning solution that aims at reducing the set prediction size.
It builds on cross-validation-based CP, rather than the less efficient validation-based CP.
It preserves formal per-task calibration guarantees, rather than less stringent task-marginal guarantees.
arXiv Detail & Related papers (2022-10-06T17:21:03Z) - Unsupervised Calibration under Covariate Shift [92.02278658443166]
We introduce the problem of calibration under domain shift and propose an importance sampling based approach to address it.
We evaluate and discuss the efficacy of our method on both real-world datasets and synthetic datasets.
arXiv Detail & Related papers (2020-06-29T21:50:07Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.