Does Confidence Calibration Help Conformal Prediction?
- URL: http://arxiv.org/abs/2402.04344v1
- Date: Tue, 6 Feb 2024 19:27:48 GMT
- Title: Does Confidence Calibration Help Conformal Prediction?
- Authors: Huajun Xi, Jianguo Huang, Lei Feng, Hongxin Wei
- Abstract summary: We show that post-hoc calibration methods lead to larger prediction sets with improved calibration.
We propose a novel method, $textbfConformal Temperature Scaling$ (ConfTS), which rectifies the objective through the gap between the threshold and the non-conformity score of the ground-truth label.
- Score: 12.119612461168941
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Conformal prediction, as an emerging uncertainty qualification technique,
constructs prediction sets that are guaranteed to contain the true label with
high probability. Previous works usually employ temperature scaling to
calibrate the classifier, assuming that confidence calibration can benefit
conformal prediction. In this work, we first show that post-hoc calibration
methods surprisingly lead to larger prediction sets with improved calibration,
while over-confidence with small temperatures benefits the conformal prediction
performance instead. Theoretically, we prove that high confidence reduces the
probability of appending a new class in the prediction set. Inspired by the
analysis, we propose a novel method, $\textbf{Conformal Temperature Scaling}$
(ConfTS), which rectifies the objective through the gap between the threshold
and the non-conformity score of the ground-truth label. In this way, the new
objective of ConfTS will optimize the temperature value toward an optimal set
that satisfies the $\textit{marginal coverage}$. Experiments demonstrate that
our method can effectively improve widely-used conformal prediction methods.
Related papers
- On Temperature Scaling and Conformal Prediction of Deep Classifiers [9.975341265604577]
Two popular approaches for that aim are: 1): modifies the classifier's softmax values such that the maximal value better estimates the correctness probability; and 2) Conformal Prediction (CP): produces a prediction set of candidate labels that contains the true label with a user-specified probability.
In practice, both types of indications are desirable, yet, so far the interplay between them has not been investigated.
arXiv Detail & Related papers (2024-02-08T16:45:12Z) - Equal Opportunity of Coverage in Fair Regression [50.76908018786335]
We study fair machine learning (ML) under predictive uncertainty to enable reliable and trustworthy decision-making.
We propose Equal Opportunity of Coverage (EOC) that aims to achieve two properties: (1) coverage rates for different groups with similar outcomes are close, and (2) the coverage rate for the entire population remains at a predetermined level.
arXiv Detail & Related papers (2023-11-03T21:19:59Z) - PAC Prediction Sets Under Label Shift [52.30074177997787]
Prediction sets capture uncertainty by predicting sets of labels rather than individual labels.
We propose a novel algorithm for constructing prediction sets with PAC guarantees in the label shift setting.
We evaluate our approach on five datasets.
arXiv Detail & Related papers (2023-10-19T17:57:57Z) - Conformal Prediction for Deep Classifier via Label Ranking [29.784336674173616]
Conformal prediction is a statistical framework that generates prediction sets with a desired coverage guarantee.
We propose a novel algorithm named $textitSorted Adaptive Prediction Sets$ (SAPS)
SAPS discards all the probability values except for the maximum softmax probability.
arXiv Detail & Related papers (2023-10-10T08:54:14Z) - Rethinking Confidence Calibration for Failure Prediction [37.43981354073841]
Modern deep neural networks are often overconfident for their incorrect predictions.
We find that most confidence calibration methods are useless or harmful for failure prediction.
We propose a simple hypothesis: flat minima is beneficial for failure prediction.
arXiv Detail & Related papers (2023-03-06T08:54:18Z) - Predictive Inference with Feature Conformal Prediction [80.77443423828315]
We propose feature conformal prediction, which extends the scope of conformal prediction to semantic feature spaces.
From a theoretical perspective, we demonstrate that feature conformal prediction provably outperforms regular conformal prediction under mild assumptions.
Our approach could be combined with not only vanilla conformal prediction, but also other adaptive conformal prediction methods.
arXiv Detail & Related papers (2022-10-01T02:57:37Z) - Sample-dependent Adaptive Temperature Scaling for Improved Calibration [95.7477042886242]
Post-hoc approach to compensate for neural networks being wrong is to perform temperature scaling.
We propose to predict a different temperature value for each input, allowing us to adjust the mismatch between confidence and accuracy.
We test our method on the ResNet50 and WideResNet28-10 architectures using the CIFAR10/100 and Tiny-ImageNet datasets.
arXiv Detail & Related papers (2022-07-13T14:13:49Z) - Revisiting Calibration for Question Answering [16.54743762235555]
We argue that the traditional evaluation of calibration does not reflect usefulness of the model confidence.
We propose a new calibration metric, MacroCE, that better captures whether the model assigns low confidence to wrong predictions and high confidence to correct predictions.
arXiv Detail & Related papers (2022-05-25T05:49:56Z) - T-Cal: An optimal test for the calibration of predictive models [49.11538724574202]
We consider detecting mis-calibration of predictive models using a finite validation dataset as a hypothesis testing problem.
detecting mis-calibration is only possible when the conditional probabilities of the classes are sufficiently smooth functions of the predictions.
We propose T-Cal, a minimax test for calibration based on a de-biased plug-in estimator of the $ell$-Expected Error (ECE)
arXiv Detail & Related papers (2022-03-03T16:58:54Z) - Parameterized Temperature Scaling for Boosting the Expressive Power in
Post-Hoc Uncertainty Calibration [57.568461777747515]
We introduce a novel calibration method, Parametrized Temperature Scaling (PTS)
We demonstrate that the performance of accuracy-preserving state-of-the-art post-hoc calibrators is limited by their intrinsic expressive power.
We show with extensive experiments that our novel accuracy-preserving approach consistently outperforms existing algorithms across a large number of model architectures, datasets and metrics.
arXiv Detail & Related papers (2021-02-24T10:18:30Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.