Cross-Validation Conformal Risk Control

• URL: http://arxiv.org/abs/2401.11974v2
• Date: Wed, 1 May 2024 15:33:36 GMT
• Title: Cross-Validation Conformal Risk Control
• Authors: Kfir M. Cohen, Sangwoo Park, Osvaldo Simeone, Shlomo Shamai,
• Abstract summary: Conformal risk control (CRC) is a recently proposed technique that applies post-hoc to a conventional point predictor to provide calibration guarantees. In this paper, a novel CRC method is introduced that is based on cross-validation, rather than on validation as the original CRC. CV-CRC is proved to offer theoretical guarantees on the average risk of the set predictor.
• Score: 40.2365781482563
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Conformal risk control (CRC) is a recently proposed technique that applies post-hoc to a conventional point predictor to provide calibration guarantees. Generalizing conformal prediction (CP), with CRC, calibration is ensured for a set predictor that is extracted from the point predictor to control a risk function such as the probability of miscoverage or the false negative rate. The original CRC requires the available data set to be split between training and validation data sets. This can be problematic when data availability is limited, resulting in inefficient set predictors. In this paper, a novel CRC method is introduced that is based on cross-validation, rather than on validation as the original CRC. The proposed cross-validation CRC (CV-CRC) extends a version of the jackknife-minmax from CP to CRC, allowing for the control of a broader range of risk functions. CV-CRC is proved to offer theoretical guarantees on the average risk of the set predictor. Furthermore, numerical experiments show that CV-CRC can reduce the average set size with respect to CRC when the available data are limited.

Related papers

• Anytime-Valid Conformal Risk Control [9.475553038511336]
  conformal prediction and risk control can produce prediction sets that exhibit statistically valid error control in a computationally efficient manner.<n>We extend the control to remain valid with high probability over a cumulatively growing calibration dataset at any time point.
  arXiv  Detail & Related papers  (2026-02-04T09:39:36Z)
• Selective Conformal Risk Control [0.38106124526091323]
  We propose textitSelective Conformal Risk Control (SCRC), a unified framework that integrates conformal prediction with selective classification.<n>The framework formulates uncertainty control as a two-stage problem: the first stage selects confident samples for prediction, and the second stage applies conformal risk control on the selected subset to construct calibrated prediction sets.<n>Experiments show that both methods achieve the target coverage and risk levels, with nearly identical performance, while SCRC-I exhibits slightly more conservative risk control but superior computational practicality.
  arXiv  Detail & Related papers  (2025-12-14T21:18:28Z)
• LEC: Linear Expectation Constraints for False-Discovery Control in Selective Prediction and Routing Systems [95.35293543918762]
  Large language models (LLMs) often generate unreliable answers, while uncertainty methods fail to fully distinguish correct from incorrect predictions.<n>We address this issue through the lens of false discovery rate (FDR) control, ensuring that among all accepted predictions, the proportion of errors does not exceed a target risk level.<n>We propose LEC, which reinterprets selective prediction as a constrained decision problem by enforcing a Linear Expectation Constraint.
  arXiv  Detail & Related papers  (2025-12-01T11:27:09Z)
• Conformal Lesion Segmentation for 3D Medical Images [82.92159832699583]
  We propose a risk-constrained framework that calibrates data-driven thresholds via conformalization to ensure the test-time FNR remains below a target tolerance.<n>We validate the statistical soundness and predictive performance of CLS on six 3D-LS datasets across five backbone models, and conclude with actionable insights for deploying risk-aware segmentation in clinical practice.
  arXiv  Detail & Related papers  (2025-10-19T08:21:00Z)
• Conformal Risk Training: End-to-End Optimization of Conformal Risk Control [41.45834526675908]
  We introduce "conformal risk training," an end-to-end approach that differentiates through conformal OCE risk control during model training or fine-tuning.<n>Our method achieves provable risk guarantees while demonstrating significantly improved average-case performance over post-hoc approaches.
  arXiv  Detail & Related papers  (2025-10-09T19:05:45Z)
• COIN: Uncertainty-Guarding Selective Question Answering for Foundation Models with Provable Risk Guarantees [51.5976496056012]
  COIN is an uncertainty-guarding selection framework that calibrates statistically valid thresholds to filter a single generated answer per question.<n>COIN estimates the empirical error rate on a calibration set and applies confidence interval methods to establish a high-probability upper bound on the true error rate.<n>We demonstrate COIN's robustness in risk control, strong test-time power in retaining admissible answers, and predictive efficiency under limited calibration data.
  arXiv  Detail & Related papers  (2025-06-25T07:04:49Z)
• Generalization and Informativeness of Weighted Conformal Risk Control Under Covariate Shift [40.43703709267958]
  Weighted conformal risk control (W-CRC) uses data collected during the training phase to convert point predictions into prediction sets with valid risk guarantees at test time. While W-CRC provides statistical reliability, its efficiency -- measured by the size of the prediction sets -- can only be assessed at test time.
  arXiv  Detail & Related papers  (2025-01-20T11:26:36Z)
• Conformal Distributed Remote Inference in Sensor Networks Under Reliability and Communication Constraints [61.62410595953275]
  Communication-constrained distributed conformal risk control (CD-CRC) CD-CRC is a novel decision-making framework for sensor networks under communication constraints.
  arXiv  Detail & Related papers  (2024-09-12T10:12:43Z)
• Risk and cross validation in ridge regression with correlated samples [72.59731158970894]
  We provide training examples for the in- and out-of-sample risks of ridge regression when the data points have arbitrary correlations. We further extend our analysis to the case where the test point has non-trivial correlations with the training set, setting often encountered in time series forecasting. We validate our theory across a variety of high dimensional data.
  arXiv  Detail & Related papers  (2024-08-08T17:27:29Z)
• Localized Adaptive Risk Control [35.88667386998423]
  Localized Adaptive Risk Control (L-ARC) is an online calibration scheme that targets statistical localized risk guarantees. L-ARC is demonstrated via experiments to produce prediction sets with risk guarantees across different data subpopulations.
  arXiv  Detail & Related papers  (2024-05-13T17:48:45Z)
• CAP: A General Algorithm for Online Selective Conformal Prediction with FCR Control [4.137346786534721]
  It is important to control the real-time false coverage-statement rate (FCR) which measures the overall miscoverage level. We develop a general framework named CAP that performs an adaptive pick rule on historical data to construct a calibration set. We prove that CAP can achieve an exact selection-conditional coverage guarantee in the finite-sample and distribution-free regimes.
  arXiv  Detail & Related papers  (2024-03-12T15:07:20Z)
• Forking Uncertainties: Reliable Prediction and Model Predictive Control with Sequence Models via Conformal Risk Control [40.918012779935246]
  We introduce a novel post-hoc calibration procedure that operates on the predictions produced by any pre-designed probabilistic forecaster to yield reliable error bars. Unlike the state of the art, PTS-CRC can satisfy reliability definitions beyond coverage. We experimentally validate the performance of PTS-CRC prediction and control by studying a number of use cases in the context of wireless networking.
  arXiv  Detail & Related papers  (2023-10-16T11:35:41Z)
• Safe Deployment for Counterfactual Learning to Rank with Exposure-Based Risk Minimization [63.93275508300137]
  We introduce a novel risk-aware Counterfactual Learning To Rank method with theoretical guarantees for safe deployment. Our experimental results demonstrate the efficacy of our proposed method, which is effective at avoiding initial periods of bad performance when little data is available.
  arXiv  Detail & Related papers  (2023-04-26T15:54:23Z)
• Bayes risk CTC: Controllable CTC alignment in Sequence-to-Sequence tasks [63.189632935619535]
  Bayes risk CTC (BRCTC) is proposed to enforce the desired characteristics of the predicted alignment. By using BRCTC with another preference for early emissions, we obtain an improved performance-latency trade-off for online models.
  arXiv  Detail & Related papers  (2022-10-14T03:55:36Z)
• Few-Shot Calibration of Set Predictors via Meta-Learned Cross-Validation-Based Conformal Prediction [33.33774397643919]
  This paper introduces a novel meta-learning solution that aims at reducing the set prediction size. It builds on cross-validation-based CP, rather than the less efficient validation-based CP. It preserves formal per-task calibration guarantees, rather than less stringent task-marginal guarantees.
  arXiv  Detail & Related papers  (2022-10-06T17:21:03Z)
• Unsupervised Calibration under Covariate Shift [92.02278658443166]
  We introduce the problem of calibration under domain shift and propose an importance sampling based approach to address it. We evaluate and discuss the efficacy of our method on both real-world datasets and synthetic datasets.
  arXiv  Detail & Related papers  (2020-06-29T21:50:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.