Uncertainty Sets for Image Classifiers using Conformal Prediction
- URL: http://arxiv.org/abs/2009.14193v5
- Date: Sat, 3 Sep 2022 05:45:19 GMT
- Title: Uncertainty Sets for Image Classifiers using Conformal Prediction
- Authors: Anastasios Angelopoulos, Stephen Bates, Jitendra Malik, Michael I.
Jordan
- Abstract summary: We present an algorithm that modifies any classifier to output a predictive set containing the true label with a user-specified probability, such as 90%.
The algorithm is simple and fast like Platt scaling, but provides a formal finite-sample coverage guarantee for every model and dataset.
Our method modifies an existing conformal prediction algorithm to give more stable predictive sets by regularizing the small scores of unlikely classes after Platt scaling.
- Score: 112.54626392838163
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Convolutional image classifiers can achieve high predictive accuracy, but
quantifying their uncertainty remains an unresolved challenge, hindering their
deployment in consequential settings. Existing uncertainty quantification
techniques, such as Platt scaling, attempt to calibrate the network's
probability estimates, but they do not have formal guarantees. We present an
algorithm that modifies any classifier to output a predictive set containing
the true label with a user-specified probability, such as 90%. The algorithm is
simple and fast like Platt scaling, but provides a formal finite-sample
coverage guarantee for every model and dataset. Our method modifies an existing
conformal prediction algorithm to give more stable predictive sets by
regularizing the small scores of unlikely classes after Platt scaling. In
experiments on both Imagenet and Imagenet-V2 with ResNet-152 and other
classifiers, our scheme outperforms existing approaches, achieving coverage
with sets that are often factors of 5 to 10 smaller than a stand-alone Platt
scaling baseline.
Related papers
- Trustworthy Classification through Rank-Based Conformal Prediction Sets [9.559062601251464]
We propose a novel conformal prediction method that employs a rank-based score function suitable for classification models.
Our approach constructs prediction sets that achieve the desired coverage rate while managing their size.
Our contributions include a novel conformal prediction method, theoretical analysis, and empirical evaluation.
arXiv Detail & Related papers (2024-07-05T10:43:41Z) - Provably Robust Conformal Prediction with Improved Efficiency [29.70455766394585]
Conformal prediction is a powerful tool to generate uncertainty sets with guaranteed coverage.
adversarial examples are able to manipulate conformal methods to construct prediction sets with invalid coverage rates.
We propose two novel methods, Post-Training Transformation (PTT) and Robust Conformal Training (RCT), to effectively reduce prediction set size with little overhead.
arXiv Detail & Related papers (2024-04-30T15:49:01Z) - Domain-adaptive and Subgroup-specific Cascaded Temperature Regression
for Out-of-distribution Calibration [16.930766717110053]
We propose a novel meta-set-based cascaded temperature regression method for post-hoc calibration.
We partition each meta-set into subgroups based on predicted category and confidence level, capturing diverse uncertainties.
A regression network is then trained to derive category-specific and confidence-level-specific scaling, achieving calibration across meta-sets.
arXiv Detail & Related papers (2024-02-14T14:35:57Z) - PAC Prediction Sets Under Label Shift [52.30074177997787]
Prediction sets capture uncertainty by predicting sets of labels rather than individual labels.
We propose a novel algorithm for constructing prediction sets with PAC guarantees in the label shift setting.
We evaluate our approach on five datasets.
arXiv Detail & Related papers (2023-10-19T17:57:57Z) - NUQ: Nonparametric Uncertainty Quantification for Deterministic Neural
Networks [151.03112356092575]
We show the principled way to measure the uncertainty of predictions for a classifier based on Nadaraya-Watson's nonparametric estimate of the conditional label distribution.
We demonstrate the strong performance of the method in uncertainty estimation tasks on a variety of real-world image datasets.
arXiv Detail & Related papers (2022-02-07T12:30:45Z) - Almost Tight L0-norm Certified Robustness of Top-k Predictions against
Adversarial Perturbations [78.23408201652984]
Top-k predictions are used in many real-world applications such as machine learning as a service, recommender systems, and web searches.
Our work is based on randomized smoothing, which builds a provably robust classifier via randomizing an input.
For instance, our method can build a classifier that achieves a certified top-3 accuracy of 69.2% on ImageNet when an attacker can arbitrarily perturb 5 pixels of a testing image.
arXiv Detail & Related papers (2020-11-15T21:34:44Z) - Knowing what you know: valid and validated confidence sets in multiclass
and multilabel prediction [0.8594140167290097]
We develop conformal prediction methods for constructing valid confidence sets in multiclass and multilabel problems.
By leveraging ideas from quantile regression, we build methods that always guarantee correct coverage but additionally provide conditional coverage for both multiclass and multilabel prediction problems.
arXiv Detail & Related papers (2020-04-21T17:45:38Z) - Meta-Learned Confidence for Few-shot Learning [60.6086305523402]
A popular transductive inference technique for few-shot metric-based approaches, is to update the prototype of each class with the mean of the most confident query examples.
We propose to meta-learn the confidence for each query sample, to assign optimal weights to unlabeled queries.
We validate our few-shot learning model with meta-learned confidence on four benchmark datasets.
arXiv Detail & Related papers (2020-02-27T10:22:17Z) - Certified Robustness to Label-Flipping Attacks via Randomized Smoothing [105.91827623768724]
Machine learning algorithms are susceptible to data poisoning attacks.
We present a unifying view of randomized smoothing over arbitrary functions.
We propose a new strategy for building classifiers that are pointwise-certifiably robust to general data poisoning attacks.
arXiv Detail & Related papers (2020-02-07T21:28:30Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.