Validity, consonant plausibility measures, and conformal prediction
- URL: http://arxiv.org/abs/2001.09225v3
- Date: Thu, 9 Jun 2022 19:17:40 GMT
- Title: Validity, consonant plausibility measures, and conformal prediction
- Authors: Leonardo Cella and Ryan Martin
- Abstract summary: We present a new notion, called Type-2 validity, relevant to other prediction-related tasks.
We show that both types of prediction validity can be achieved by interpreting the conformal prediction output as the contour function of a consonant plausibility measure.
- Score: 7.563864405505623
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Prediction of future observations is an important and challenging problem.
The two mainstream approaches for quantifying prediction uncertainty use
prediction regions and predictive distributions, respectively, with the latter
believed to be more informative because it can perform other prediction-related
tasks. The standard notion of validity, what we refer to here as Type-1
validity, focuses on coverage probability of prediction regions, while a notion
of validity relevant to the other prediction-related tasks performed by
predictive distributions is lacking. Here we present a new notion, called
Type-2 validity, relevant to these other prediction tasks. We establish
connections between Type-2 validity and coherence properties, and show that
imprecise probability considerations are required in order to achieve it. We go
on to show that both types of prediction validity can be achieved by
interpreting the conformal prediction output as the contour function of a
consonant plausibility measure. We also offer an alternative characterization
of conformal prediction, based on a new nonparametric inferential model
construction, wherein the appearance of consonance is natural, and prove its
validity.
Related papers
- Self-Calibrating Conformal Prediction [16.606421967131524]
We introduce Self-Calibrating Conformal Prediction to deliver calibrated point predictions alongside prediction intervals with finite-sample validity conditional on these predictions.
We show that our method improves calibrated interval efficiency through model calibration and offers a practical alternative to feature-conditional validity.
arXiv Detail & Related papers (2024-02-11T21:12:21Z) - Confidence and Dispersity Speak: Characterising Prediction Matrix for
Unsupervised Accuracy Estimation [51.809741427975105]
This work aims to assess how well a model performs under distribution shifts without using labels.
We use the nuclear norm that has been shown to be effective in characterizing both properties.
We show that the nuclear norm is more accurate and robust in accuracy than existing methods.
arXiv Detail & Related papers (2023-02-02T13:30:48Z) - Predictive Inference with Feature Conformal Prediction [80.77443423828315]
We propose feature conformal prediction, which extends the scope of conformal prediction to semantic feature spaces.
From a theoretical perspective, we demonstrate that feature conformal prediction provably outperforms regular conformal prediction under mild assumptions.
Our approach could be combined with not only vanilla conformal prediction, but also other adaptive conformal prediction methods.
arXiv Detail & Related papers (2022-10-01T02:57:37Z) - Uncertainty estimation of pedestrian future trajectory using Bayesian
approximation [137.00426219455116]
Under dynamic traffic scenarios, planning based on deterministic predictions is not trustworthy.
The authors propose to quantify uncertainty during forecasting using approximation which deterministic approaches fail to capture.
The effect of dropout weights and long-term prediction on future state uncertainty has been studied.
arXiv Detail & Related papers (2022-05-04T04:23:38Z) - Dense Uncertainty Estimation [62.23555922631451]
In this paper, we investigate neural networks and uncertainty estimation techniques to achieve both accurate deterministic prediction and reliable uncertainty estimation.
We work on two types of uncertainty estimations solutions, namely ensemble based methods and generative model based methods, and explain their pros and cons while using them in fully/semi/weakly-supervised framework.
arXiv Detail & Related papers (2021-10-13T01:23:48Z) - CovarianceNet: Conditional Generative Model for Correct Covariance
Prediction in Human Motion Prediction [71.31516599226606]
We present a new method to correctly predict the uncertainty associated with the predicted distribution of future trajectories.
Our approach, CovariaceNet, is based on a Conditional Generative Model with Gaussian latent variables.
arXiv Detail & Related papers (2021-09-07T09:38:24Z) - When in Doubt: Neural Non-Parametric Uncertainty Quantification for
Epidemic Forecasting [70.54920804222031]
Most existing forecasting models disregard uncertainty quantification, resulting in mis-calibrated predictions.
Recent works in deep neural models for uncertainty-aware time-series forecasting also have several limitations.
We model the forecasting task as a probabilistic generative process and propose a functional neural process model called EPIFNP.
arXiv Detail & Related papers (2021-06-07T18:31:47Z) - Approximation to Object Conditional Validity with Conformal Predictors [0.2538209532048866]
Conformal predictors are machine learning algorithms that output prediction intervals that have a guarantee of marginal validity for finite samples.
It has been shown that such conditional validity is impossible to guarantee for non-trivial prediction problems for finite samples.
In this article, instead of trying to achieve a strong conditional validity result, the weaker goal of achieving an approximation to conditional validity is considered.
arXiv Detail & Related papers (2021-02-15T10:14:44Z) - Conformal e-prediction [0.0]
Conformal e-prediction is conceptually simpler and had been developed in the 1990s as precursor of conformal prediction.
This paper re-examines relations between conformal prediction and conformal e-prediction systematically from a modern perspective.
arXiv Detail & Related papers (2020-01-16T18:41:17Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.