Distribution-Free Finite-Sample Guarantees and Split Conformal
Prediction
- URL: http://arxiv.org/abs/2210.14735v1
- Date: Wed, 26 Oct 2022 14:12:24 GMT
- Title: Distribution-Free Finite-Sample Guarantees and Split Conformal
Prediction
- Authors: Roel Hulsman
- Abstract summary: split conformal prediction represents a promising avenue to obtain finite-sample guarantees under minimal distribution-free assumptions.
We highlight the connection between split conformal prediction and classical tolerance predictors developed in the 1940s.
- Score: 0.0
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Modern black-box predictive models are often accompanied by weak performance
guarantees that only hold asymptotically in the size of the dataset or require
strong parametric assumptions. In response to this, split conformal prediction
represents a promising avenue to obtain finite-sample guarantees under minimal
distribution-free assumptions. Although prediction set validity most often
concerns marginal coverage, we explore the related but different guarantee of
tolerance regions, reformulating known results in the language of nested
prediction sets and extending on the duality between marginal coverage and
tolerance regions. Furthermore, we highlight the connection between split
conformal prediction and classical tolerance predictors developed in the 1940s,
as well as recent developments in distribution-free risk control. One result
that transfers from classical tolerance predictors is that the coverage of a
prediction set based on order statistics, conditional on the calibration set,
is a random variable stochastically dominating the Beta distribution. We
demonstrate the empirical effectiveness of our findings on synthetic and real
datasets using a popular split conformal prediction procedure called
conformalized quantile regression (CQR).
Related papers
- Conditionally valid Probabilistic Conformal Prediction [57.80927226809277]
We develop a new method for creating prediction sets that combines the flexibility of conformal methods with an estimate of the conditional distribution.
We demonstrate the effectiveness of our approach through extensive simulations, showing that it outperforms existing methods in terms of conditional coverage.
arXiv Detail & Related papers (2024-07-01T20:44:48Z) - Robust Conformal Prediction Using Privileged Information [17.886554223172517]
We develop a method to generate prediction sets with a guaranteed coverage rate that is robust to corruptions in the training data.
Our approach builds on conformal prediction, a powerful framework to construct prediction sets that are valid under the i.i.d assumption.
arXiv Detail & Related papers (2024-06-08T08:56:47Z) - Distribution-Free Conformal Joint Prediction Regions for Neural Marked Temporal Point Processes [4.324839843326325]
We develop more reliable methods for uncertainty in neural TPP models via the framework of conformal prediction.
A primary objective is to generate a distribution-free joint prediction region for an event's arrival time and mark, with a finite-sample marginal coverage guarantee.
arXiv Detail & Related papers (2024-01-09T15:28:29Z) - When Rigidity Hurts: Soft Consistency Regularization for Probabilistic
Hierarchical Time Series Forecasting [69.30930115236228]
Probabilistic hierarchical time-series forecasting is an important variant of time-series forecasting.
Most methods focus on point predictions and do not provide well-calibrated probabilistic forecasts distributions.
We propose PROFHiT, a fully probabilistic hierarchical forecasting model that jointly models forecast distribution of entire hierarchy.
arXiv Detail & Related papers (2023-10-17T20:30:16Z) - Conditional validity of heteroskedastic conformal regression [12.905195278168506]
Conformal prediction and split conformal prediction offer a distribution-free approach to estimating prediction intervals with statistical guarantees.
Recent work has shown that split conformal prediction can produce state-of-the-art prediction intervals when focusing on marginal coverage.
This paper tries to shed new light on how prediction intervals can be constructed, using methods such as normalized and Mondrian conformal prediction.
arXiv Detail & Related papers (2023-09-15T11:10:46Z) - Conformal Prediction for Federated Uncertainty Quantification Under
Label Shift [57.54977668978613]
Federated Learning (FL) is a machine learning framework where many clients collaboratively train models.
We develop a new conformal prediction method based on quantile regression and take into account privacy constraints.
arXiv Detail & Related papers (2023-06-08T11:54:58Z) - Conformal Prediction with Missing Values [19.18178194789968]
We first show that the marginal coverage guarantee of conformal prediction holds on imputed data for any missingness distribution.
We then show that a universally consistent quantile regression algorithm trained on the imputed data is Bayes optimal for the pinball risk.
arXiv Detail & Related papers (2023-06-05T09:28:03Z) - Post-selection Inference for Conformal Prediction: Trading off Coverage
for Precision [0.0]
Traditionally, conformal prediction inference requires a data-independent specification of miscoverage level.
We develop simultaneous conformal inference to account for data-dependent miscoverage levels.
arXiv Detail & Related papers (2023-04-12T20:56:43Z) - Predictive Inference with Feature Conformal Prediction [80.77443423828315]
We propose feature conformal prediction, which extends the scope of conformal prediction to semantic feature spaces.
From a theoretical perspective, we demonstrate that feature conformal prediction provably outperforms regular conformal prediction under mild assumptions.
Our approach could be combined with not only vanilla conformal prediction, but also other adaptive conformal prediction methods.
arXiv Detail & Related papers (2022-10-01T02:57:37Z) - CovarianceNet: Conditional Generative Model for Correct Covariance
Prediction in Human Motion Prediction [71.31516599226606]
We present a new method to correctly predict the uncertainty associated with the predicted distribution of future trajectories.
Our approach, CovariaceNet, is based on a Conditional Generative Model with Gaussian latent variables.
arXiv Detail & Related papers (2021-09-07T09:38:24Z) - Private Prediction Sets [72.75711776601973]
Machine learning systems need reliable uncertainty quantification and protection of individuals' privacy.
We present a framework that treats these two desiderata jointly.
We evaluate the method on large-scale computer vision datasets.
arXiv Detail & Related papers (2021-02-11T18:59:11Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.