On the Expected Size of Conformal Prediction Sets
- URL: http://arxiv.org/abs/2306.07254v3
- Date: Sat, 9 Mar 2024 03:20:06 GMT
- Title: On the Expected Size of Conformal Prediction Sets
- Authors: Guneet S. Dhillon and George Deligiannidis and Tom Rainforth
- Abstract summary: We theoretically quantify the expected size of the prediction sets under the split conformal prediction framework.
As this precise formulation cannot usually be calculated directly, we derive point estimates and high-probability bounds interval.
We corroborate the efficacy of our results with experiments on real-world datasets for both regression and classification problems.
- Score: 24.161372736642157
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: While conformal predictors reap the benefits of rigorous statistical
guarantees on their error frequency, the size of their corresponding prediction
sets is critical to their practical utility. Unfortunately, there is currently
a lack of finite-sample analysis and guarantees for their prediction set sizes.
To address this shortfall, we theoretically quantify the expected size of the
prediction sets under the split conformal prediction framework. As this precise
formulation cannot usually be calculated directly, we further derive point
estimates and high-probability interval bounds that can be empirically
computed, providing a practical method for characterizing the expected set
size. We corroborate the efficacy of our results with experiments on real-world
datasets for both regression and classification problems.
Related papers
- Generalization and Informativeness of Conformal Prediction [36.407171992845456]
Con conformal prediction (CP) transforms an arbitrary base predictor into a set predictor with coverage guarantees.
CP certifies the predicted set to contain the target quantity with a user-defined tolerance, but it does not provide control over the average size of the predicted sets.
A theoretical connection is established between the generalization properties of the base predictor and the informativeness of the resulting CP prediction sets.
The derived upper bound provides insights into the dependence of the average size of the CP set predictor on the amount of calibration data, the target reliability, and the generalization performance of the base predictor.
arXiv Detail & Related papers (2024-01-22T10:14:45Z) - Improving Adaptive Conformal Prediction Using Self-Supervised Learning [72.2614468437919]
We train an auxiliary model with a self-supervised pretext task on top of an existing predictive model and use the self-supervised error as an additional feature to estimate nonconformity scores.
We empirically demonstrate the benefit of the additional information using both synthetic and real data on the efficiency (width), deficit, and excess of conformal prediction intervals.
arXiv Detail & Related papers (2023-02-23T18:57:14Z) - Distribution-Free Finite-Sample Guarantees and Split Conformal
Prediction [0.0]
split conformal prediction represents a promising avenue to obtain finite-sample guarantees under minimal distribution-free assumptions.
We highlight the connection between split conformal prediction and classical tolerance predictors developed in the 1940s.
arXiv Detail & Related papers (2022-10-26T14:12:24Z) - Predictive Inference with Feature Conformal Prediction [80.77443423828315]
We propose feature conformal prediction, which extends the scope of conformal prediction to semantic feature spaces.
From a theoretical perspective, we demonstrate that feature conformal prediction provably outperforms regular conformal prediction under mild assumptions.
Our approach could be combined with not only vanilla conformal prediction, but also other adaptive conformal prediction methods.
arXiv Detail & Related papers (2022-10-01T02:57:37Z) - Conformal prediction set for time-series [16.38369532102931]
Uncertainty quantification is essential to studying complex machine learning methods.
We develop Ensemble Regularized Adaptive Prediction Set (ERAPS) to construct prediction sets for time-series.
We show valid marginal and conditional coverage by ERAPS, which also tends to yield smaller prediction sets than competing methods.
arXiv Detail & Related papers (2022-06-15T23:48:53Z) - Dense Uncertainty Estimation [62.23555922631451]
In this paper, we investigate neural networks and uncertainty estimation techniques to achieve both accurate deterministic prediction and reliable uncertainty estimation.
We work on two types of uncertainty estimations solutions, namely ensemble based methods and generative model based methods, and explain their pros and cons while using them in fully/semi/weakly-supervised framework.
arXiv Detail & Related papers (2021-10-13T01:23:48Z) - Unifying Lower Bounds on Prediction Dimension of Consistent Convex
Surrogates [12.751555473216683]
Given a prediction task, understanding when one can and cannot design a consistent convex surrogate loss is an important area of machine learning research.
We unify these settings using tools from property elicitation, and give a general lower bound on prediction dimension.
Our lower bound tightens existing results in the case of discrete predictions, showing that previous calibration-based bounds can largely be recovered via property elicitation.
For continuous estimation, our lower bound resolves on open problem on estimating measures of risk and uncertainty.
arXiv Detail & Related papers (2021-02-16T15:29:05Z) - Private Prediction Sets [72.75711776601973]
Machine learning systems need reliable uncertainty quantification and protection of individuals' privacy.
We present a framework that treats these two desiderata jointly.
We evaluate the method on large-scale computer vision datasets.
arXiv Detail & Related papers (2021-02-11T18:59:11Z) - Conformal prediction for time series [16.38369532102931]
textttEnbPI wraps around ensemble predictors, which is closely related to conformal prediction (CP) but does not require data exchangeability.
We perform extensive simulation and real-data analyses to demonstrate its effectiveness compared with existing methods.
arXiv Detail & Related papers (2020-10-18T21:05:32Z) - Counterfactual Predictions under Runtime Confounding [74.90756694584839]
We study the counterfactual prediction task in the setting where all relevant factors are captured in the historical data.
We propose a doubly-robust procedure for learning counterfactual prediction models in this setting.
arXiv Detail & Related papers (2020-06-30T15:49:05Z) - Performance metrics for intervention-triggering prediction models do not
reflect an expected reduction in outcomes from using the model [71.9860741092209]
Clinical researchers often select among and evaluate risk prediction models.
Standard metrics calculated from retrospective data are only related to model utility under certain assumptions.
When predictions are delivered repeatedly throughout time, the relationship between standard metrics and utility is further complicated.
arXiv Detail & Related papers (2020-06-02T16:26:49Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.