Optimized conformal classification using gradient descent approximation
- URL: http://arxiv.org/abs/2105.11255v1
- Date: Mon, 24 May 2021 13:14:41 GMT
- Title: Optimized conformal classification using gradient descent approximation
- Authors: Anthony Bellotti
- Abstract summary: Conformal predictors allow predictions to be made with a user-defined confidence level.
We consider an approach to train the conformal predictor directly with maximum predictive efficiency.
We test the method on several real world data sets and find that the method is promising.
- Score: 0.2538209532048866
- License: http://creativecommons.org/licenses/by-nc-nd/4.0/
- Abstract: Conformal predictors are an important class of algorithms that allow
predictions to be made with a user-defined confidence level. They are able to
do this by outputting prediction sets, rather than simple point predictions.
The conformal predictor is valid in the sense that the accuracy of its
predictions is guaranteed to meet the confidence level, only assuming
exchangeability in the data. Since accuracy is guaranteed, the performance of a
conformal predictor is measured through the efficiency of the prediction sets.
Typically, a conformal predictor is built on an underlying machine learning
algorithm and hence its predictive power is inherited from this algorithm.
However, since the underlying machine learning algorithm is not trained with
the objective of minimizing predictive efficiency it means that the resulting
conformal predictor may be sub-optimal and not aligned sufficiently to this
objective. Hence, in this study we consider an approach to train the conformal
predictor directly with maximum predictive efficiency as the optimization
objective, and we focus specifically on the inductive conformal predictor for
classification. To do this, the conformal predictor is approximated by a
differentiable objective function and gradient descent used to optimize it. The
resulting parameter estimates are then passed to a proper inductive conformal
predictor to give valid prediction sets. We test the method on several real
world data sets and find that the method is promising and in most cases gives
improved predictive efficiency against a baseline conformal predictor.
Related papers
- Towards Human-AI Complementarity with Predictions Sets [14.071862670474832]
Decision support systems based on prediction sets have proven to be effective at helping human experts solve classification tasks.
We show that the prediction sets constructed using conformal prediction are, in general, suboptimal in terms of average accuracy.
We introduce a greedy algorithm that, for a large class of expert models and non-optimal scores, is guaranteed to find prediction sets that provably offer equal or greater performance.
arXiv Detail & Related papers (2024-05-27T18:00:00Z) - Variational Inference with Coverage Guarantees in Simulation-Based Inference [18.818573945984873]
We propose Conformalized Amortized Neural Variational Inference (CANVI)
CANVI constructs conformalized predictors based on each candidate, compares the predictors using a metric known as predictive efficiency, and returns the most efficient predictor.
We prove lower bounds on the predictive efficiency of the regions produced by CANVI and explore how the quality of a posterior approximation relates to the predictive efficiency of prediction regions based on that approximation.
arXiv Detail & Related papers (2023-05-23T17:24:04Z) - Improving Adaptive Conformal Prediction Using Self-Supervised Learning [72.2614468437919]
We train an auxiliary model with a self-supervised pretext task on top of an existing predictive model and use the self-supervised error as an additional feature to estimate nonconformity scores.
We empirically demonstrate the benefit of the additional information using both synthetic and real data on the efficiency (width), deficit, and excess of conformal prediction intervals.
arXiv Detail & Related papers (2023-02-23T18:57:14Z) - Predictive Inference with Feature Conformal Prediction [80.77443423828315]
We propose feature conformal prediction, which extends the scope of conformal prediction to semantic feature spaces.
From a theoretical perspective, we demonstrate that feature conformal prediction provably outperforms regular conformal prediction under mild assumptions.
Our approach could be combined with not only vanilla conformal prediction, but also other adaptive conformal prediction methods.
arXiv Detail & Related papers (2022-10-01T02:57:37Z) - Efficient and Differentiable Conformal Prediction with General Function
Classes [96.74055810115456]
We propose a generalization of conformal prediction to multiple learnable parameters.
We show that it achieves approximate valid population coverage and near-optimal efficiency within class.
Experiments show that our algorithm is able to learn valid prediction sets and improve the efficiency significantly.
arXiv Detail & Related papers (2022-02-22T18:37:23Z) - CovarianceNet: Conditional Generative Model for Correct Covariance
Prediction in Human Motion Prediction [71.31516599226606]
We present a new method to correctly predict the uncertainty associated with the predicted distribution of future trajectories.
Our approach, CovariaceNet, is based on a Conditional Generative Model with Gaussian latent variables.
arXiv Detail & Related papers (2021-09-07T09:38:24Z) - Private Prediction Sets [72.75711776601973]
Machine learning systems need reliable uncertainty quantification and protection of individuals' privacy.
We present a framework that treats these two desiderata jointly.
We evaluate the method on large-scale computer vision datasets.
arXiv Detail & Related papers (2021-02-11T18:59:11Z) - AutoCP: Automated Pipelines for Accurate Prediction Intervals [84.16181066107984]
This paper proposes an AutoML framework called Automatic Machine Learning for Conformal Prediction (AutoCP)
Unlike the familiar AutoML frameworks that attempt to select the best prediction model, AutoCP constructs prediction intervals that achieve the user-specified target coverage rate.
We tested AutoCP on a variety of datasets and found that it significantly outperforms benchmark algorithms.
arXiv Detail & Related papers (2020-06-24T23:13:11Z) - Fast, Optimal, and Targeted Predictions using Parametrized Decision
Analysis [0.0]
We develop a class of parametrized actions for Bayesian decision analysis that produce optimal, scalable, and simple targeted predictions.
Predictions are constructed for physical activity data from the National Health and Nutrition Examination Survey.
arXiv Detail & Related papers (2020-06-23T15:55:47Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.