Self-Calibrating Conformal Prediction

• URL: http://arxiv.org/abs/2402.07307v3
• Date: Thu, 31 Oct 2024 16:39:11 GMT
• Title: Self-Calibrating Conformal Prediction
• Authors: Lars van der Laan, Ahmed M. Alaa,
• Abstract summary: We introduce Self-Calibrating Conformal Prediction to deliver calibrated point predictions alongside prediction intervals with finite-sample validity conditional on these predictions. We show that our method improves calibrated interval efficiency through model calibration and offers a practical alternative to feature-conditional validity.
• Score: 16.606421967131524
• License:
• Abstract: In machine learning, model calibration and predictive inference are essential for producing reliable predictions and quantifying uncertainty to support decision-making. Recognizing the complementary roles of point and interval predictions, we introduce Self-Calibrating Conformal Prediction, a method that combines Venn-Abers calibration and conformal prediction to deliver calibrated point predictions alongside prediction intervals with finite-sample validity conditional on these predictions. To achieve this, we extend the original Venn-Abers procedure from binary classification to regression. Our theoretical framework supports analyzing conformal prediction methods that involve calibrating model predictions and subsequently constructing conditionally valid prediction intervals on the same data, where the conditioning set or conformity scores may depend on the calibrated predictions. Real-data experiments show that our method improves interval efficiency through model calibration and offers a practical alternative to feature-conditional validity.

Related papers

• Toward Conditional Distribution Calibration in Survival Prediction [6.868842871753991]
  We propose a method based on conformal prediction that uses the model's predicted individual survival probability at that instance's observed time. We provide theoretical guarantees for both marginal and conditional calibration and test it extensively across 15 diverse real-world datasets.
  arXiv  Detail & Related papers  (2024-10-27T20:19:46Z)
• Provably Reliable Conformal Prediction Sets in the Presence of Data Poisoning [53.42244686183879]
  Conformal prediction provides model-agnostic and distribution-free uncertainty quantification. Yet, conformal prediction is not reliable under poisoning attacks where adversaries manipulate both training and calibration data. We propose reliable prediction sets (RPS): the first efficient method for constructing conformal prediction sets with provable reliability guarantees under poisoning.
  arXiv  Detail & Related papers  (2024-10-13T15:37:11Z)
• Split Conformal Prediction under Data Contamination [14.23965125128232]
  We study the robustness of split conformal prediction in a data contamination setting. We quantify the impact of corrupted data on the coverage and efficiency of the constructed sets. We propose an adjustment in the classification setting which we call Contamination Robust Conformal Prediction.
  arXiv  Detail & Related papers  (2024-07-10T14:33:28Z)
• Conformal Approach To Gaussian Process Surrogate Evaluation With Coverage Guarantees [47.22930583160043]
  We propose a method for building adaptive cross-conformal prediction intervals. The resulting conformal prediction intervals exhibit a level of adaptivity akin to Bayesian credibility sets. The potential applicability of the method is demonstrated in the context of surrogate modeling of an expensive-to-evaluate simulator of the clogging phenomenon in steam generators of nuclear reactors.
  arXiv  Detail & Related papers  (2024-01-15T14:45:18Z)
• Conditional validity of heteroskedastic conformal regression [12.905195278168506]
  Conformal prediction and split conformal prediction offer a distribution-free approach to estimating prediction intervals with statistical guarantees. Recent work has shown that split conformal prediction can produce state-of-the-art prediction intervals when focusing on marginal coverage. This paper tries to shed new light on how prediction intervals can be constructed, using methods such as normalized and Mondrian conformal prediction.
  arXiv  Detail & Related papers  (2023-09-15T11:10:46Z)
• Improving Adaptive Conformal Prediction Using Self-Supervised Learning [72.2614468437919]
  We train an auxiliary model with a self-supervised pretext task on top of an existing predictive model and use the self-supervised error as an additional feature to estimate nonconformity scores. We empirically demonstrate the benefit of the additional information using both synthetic and real data on the efficiency (width), deficit, and excess of conformal prediction intervals.
  arXiv  Detail & Related papers  (2023-02-23T18:57:14Z)
• Predictive Inference with Feature Conformal Prediction [80.77443423828315]
  We propose feature conformal prediction, which extends the scope of conformal prediction to semantic feature spaces. From a theoretical perspective, we demonstrate that feature conformal prediction provably outperforms regular conformal prediction under mild assumptions. Our approach could be combined with not only vanilla conformal prediction, but also other adaptive conformal prediction methods.
  arXiv  Detail & Related papers  (2022-10-01T02:57:37Z)
• CovarianceNet: Conditional Generative Model for Correct Covariance Prediction in Human Motion Prediction [71.31516599226606]
  We present a new method to correctly predict the uncertainty associated with the predicted distribution of future trajectories. Our approach, CovariaceNet, is based on a Conditional Generative Model with Gaussian latent variables.
  arXiv  Detail & Related papers  (2021-09-07T09:38:24Z)
• Private Prediction Sets [72.75711776601973]
  Machine learning systems need reliable uncertainty quantification and protection of individuals' privacy. We present a framework that treats these two desiderata jointly. We evaluate the method on large-scale computer vision datasets.
  arXiv  Detail & Related papers  (2021-02-11T18:59:11Z)
• Short-term prediction of Time Series based on bounding techniques [0.0]
  This paper is reconsidered the prediction problem in time series framework by using a new non-parametric approach. The innovation is to consider both deterministic and deterministic-stochastic assumptions in order to obtain the upper bound of the prediction error. A benchmark is included to illustrate that the proposed predictor can obtain suitable results in a prediction scheme, and can be an interesting alternative method to the classical non-parametric methods.
  arXiv  Detail & Related papers  (2021-01-26T11:27:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.