Probabilistic Conformal Coverage Guarantees in Small-Data Settings

• URL: http://arxiv.org/abs/2509.15349v1
• Date: Thu, 18 Sep 2025 18:41:50 GMT
• Title: Probabilistic Conformal Coverage Guarantees in Small-Data Settings
• Authors: Petrus H. Zwart,
• Abstract summary: Conformal prediction provides distribution-free prediction sets with guaranteed marginal coverage.<n>In split conformal prediction this guarantee is training-conditional only in expectation.<n>This variance undermines effective risk control in practical applications.
• Score: 0.02648566468224904
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Conformal prediction provides distribution-free prediction sets with guaranteed marginal coverage. However, in split conformal prediction this guarantee is training-conditional only in expectation: across many calibration draws, the average coverage equals the nominal level, but the realized coverage for a single calibration set may vary substantially. This variance undermines effective risk control in practical applications. Here we introduce the Small Sample Beta Correction (SSBC), a plug-and-play adjustment to the conformal significance level that leverages the exact finite-sample distribution of conformal coverage to provide probabilistic guarantees, ensuring that with user-defined probability over the calibration draw, the deployed predictor achieves at least the desired coverage.

Related papers

• Conditional Coverage Diagnostics for Conformal Prediction [47.93989136542648]
  We show that conditional coverage estimation can be a classification problem.<n>We call the resulting family of metrics excess risk of the target coverage (ERT)<n>We release an open-source package for ERT as well as previous conditional coverage metrics.
  arXiv  Detail & Related papers  (2025-12-12T18:47:39Z)
• Distribution-informed Online Conformal Prediction [53.674678995825666]
  We propose Conformal Optimistic Prediction (COP), an online conformal prediction algorithm incorporating underlying data pattern into the update rule.<n>COP produces tighter prediction sets when predictable pattern exists, while retaining valid coverage guarantees even when estimates are inaccurate.<n>We prove that COP can achieve valid coverage and construct shorter prediction intervals than other baselines.
  arXiv  Detail & Related papers  (2025-12-08T17:51:49Z)
• Adaptive Coverage Policies in Conformal Prediction [46.298122008420414]
  We leverage advances in e-values and post-hoc conformal inference to improve conformal prediction methods.<n>We propose to optimize an adaptive coverage policy by training a neural network using a leave-one-out procedure on a calibration set.
  arXiv  Detail & Related papers  (2025-10-05T18:37:22Z)
• COIN: Uncertainty-Guarding Selective Question Answering for Foundation Models with Provable Risk Guarantees [51.5976496056012]
  COIN is an uncertainty-guarding selection framework that calibrates statistically valid thresholds to filter a single generated answer per question.<n>COIN estimates the empirical error rate on a calibration set and applies confidence interval methods to establish a high-probability upper bound on the true error rate.<n>We demonstrate COIN's robustness in risk control, strong test-time power in retaining admissible answers, and predictive efficiency under limited calibration data.
  arXiv  Detail & Related papers  (2025-06-25T07:04:49Z)
• Adaptive Temperature Scaling with Conformal Prediction [47.51764759462074]
  We propose to the best of our knowledge the first method for assigning calibrated probabilities to elements of a conformal prediction set.<n>Our approach frames this as an adaptive calibration problem, selecting an input-specific temperature parameter to match the desired coverage level.
  arXiv  Detail & Related papers  (2025-05-21T12:18:15Z)
• Sparse Activations as Conformal Predictors [19.298282860984116]
  We find a novel connection between conformal prediction and sparse softmax-like transformations.<n>We introduce new non-conformity scores for classification that make the calibration process correspond to the widely used temperature scaling method.<n>We show that the proposed method achieves competitive results in terms of coverage, efficiency, and adaptiveness.
  arXiv  Detail & Related papers  (2025-02-20T17:53:41Z)
• Conformal Prediction Sets with Improved Conditional Coverage using Trust Scores [52.92618442300405]
  It is impossible to achieve exact, distribution-free conditional coverage in finite samples.<n>We propose an alternative conformal prediction algorithm that targets coverage where it matters most.
  arXiv  Detail & Related papers  (2025-01-17T12:01:56Z)
• Probabilistic Conformal Prediction with Approximate Conditional Validity [81.30551968980143]
  We develop a new method for generating prediction sets that combines the flexibility of conformal methods with an estimate of the conditional distribution. Our method consistently outperforms existing approaches in terms of conditional coverage.
  arXiv  Detail & Related papers  (2024-07-01T20:44:48Z)
• Adaptive Conformal Regression with Jackknife+ Rescaled Scores [7.176758110912026]
  Conformal regression provides prediction intervals with global coverage guarantees, but often fails to capture local error distributions. We address this with a new adaptive method based on rescaling conformal scores with an estimate of local score distribution. Our approach ensures formal global coverage guarantees and is supported by new theoretical results on local coverage, including an a posteriori bound on any calibration score.
  arXiv  Detail & Related papers  (2023-05-31T14:32:26Z)
• Distribution-Free Finite-Sample Guarantees and Split Conformal Prediction [0.0]
  split conformal prediction represents a promising avenue to obtain finite-sample guarantees under minimal distribution-free assumptions. We highlight the connection between split conformal prediction and classical tolerance predictors developed in the 1940s.
  arXiv  Detail & Related papers  (2022-10-26T14:12:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.