Unified Uncertainty Calibration
- URL: http://arxiv.org/abs/2310.01202v2
- Date: Thu, 18 Jan 2024 22:48:17 GMT
- Title: Unified Uncertainty Calibration
- Authors: Kamalika Chaudhuri and David Lopez-Paz
- Abstract summary: We introduce emphunified uncertainty calibration (U2C), a holistic framework to combine aleatoric and uncertainty uncertainties.
U2C enables a clean learning-theoretical analysis of uncertainty estimation, and outperforms reject-or-classify across a variety of ImageNet benchmarks.
- Score: 43.733911707842005
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: To build robust, fair, and safe AI systems, we would like our classifiers to
say ``I don't know'' when facing test examples that are difficult or fall
outside of the training classes.The ubiquitous strategy to predict under
uncertainty is the simplistic \emph{reject-or-classify} rule: abstain from
prediction if epistemic uncertainty is high, classify otherwise.Unfortunately,
this recipe does not allow different sources of uncertainty to communicate with
each other, produces miscalibrated predictions, and it does not allow to
correct for misspecifications in our uncertainty estimates. To address these
three issues, we introduce \emph{unified uncertainty calibration (U2C)}, a
holistic framework to combine aleatoric and epistemic uncertainties. U2C
enables a clean learning-theoretical analysis of uncertainty estimation, and
outperforms reject-or-classify across a variety of ImageNet benchmarks. Our
code is available at:
https://github.com/facebookresearch/UnifiedUncertaintyCalibration
Related papers
- Certainly Uncertain: A Benchmark and Metric for Multimodal Epistemic and Aleatoric Awareness [106.52630978891054]
We present a taxonomy of uncertainty specific to vision-language AI systems.
We also introduce a new metric confidence-weighted accuracy, that is well correlated with both accuracy and calibration error.
arXiv Detail & Related papers (2024-07-02T04:23:54Z) - Identifying Drivers of Predictive Aleatoric Uncertainty [2.5311562666866494]
We present a simple approach to explain predictive aleatoric uncertainties.
We estimate uncertainty as predictive variance by adapting a neural network with a Gaussian output distribution.
We quantify our findings with a nuanced benchmark analysis that includes real-world datasets.
arXiv Detail & Related papers (2023-12-12T13:28:53Z) - Integrating Uncertainty Awareness into Conformalized Quantile Regression [12.875863572064986]
We propose a new variant of the Conformalized Quantile Regression (CQR) methodology to adjust quantile regressors differentially across the feature space.
Compared to CQR, our methods enjoy the same distribution-free theoretical coverage guarantees, while demonstrating stronger conditional coverage properties in simulated settings and real-world data sets alike.
arXiv Detail & Related papers (2023-06-14T18:28:53Z) - Reliability-Aware Prediction via Uncertainty Learning for Person Image
Retrieval [51.83967175585896]
UAL aims at providing reliability-aware predictions by considering data uncertainty and model uncertainty simultaneously.
Data uncertainty captures the noise" inherent in the sample, while model uncertainty depicts the model's confidence in the sample's prediction.
arXiv Detail & Related papers (2022-10-24T17:53:20Z) - What is Flagged in Uncertainty Quantification? Latent Density Models for
Uncertainty Categorization [68.15353480798244]
Uncertainty Quantification (UQ) is essential for creating trustworthy machine learning models.
Recent years have seen a steep rise in UQ methods that can flag suspicious examples.
We propose a framework for categorizing uncertain examples flagged by UQ methods in classification tasks.
arXiv Detail & Related papers (2022-07-11T19:47:00Z) - Dense Uncertainty Estimation via an Ensemble-based Conditional Latent
Variable Model [68.34559610536614]
We argue that the aleatoric uncertainty is an inherent attribute of the data and can only be correctly estimated with an unbiased oracle model.
We propose a new sampling and selection strategy at train time to approximate the oracle model for aleatoric uncertainty estimation.
Our results show that our solution achieves both accurate deterministic results and reliable uncertainty estimation.
arXiv Detail & Related papers (2021-11-22T08:54:10Z) - Recalibration of Aleatoric and Epistemic Regression Uncertainty in
Medical Imaging [2.126171264016785]
Well-calibrated uncertainty in regression allows robust rejection of unreliable predictions or detection of out-of-distribution samples.
$ sigma $ scaling is able to reliably recalibrate predictive uncertainty.
arXiv Detail & Related papers (2021-04-26T07:18:58Z) - Distribution-free uncertainty quantification for classification under
label shift [105.27463615756733]
We focus on uncertainty quantification (UQ) for classification problems via two avenues.
We first argue that label shift hurts UQ, by showing degradation in coverage and calibration.
We examine these techniques theoretically in a distribution-free framework and demonstrate their excellent practical performance.
arXiv Detail & Related papers (2021-03-04T20:51:03Z) - Getting a CLUE: A Method for Explaining Uncertainty Estimates [30.367995696223726]
We propose a novel method for interpreting uncertainty estimates from differentiable probabilistic models.
Our method, Counterfactual Latent Uncertainty Explanations (CLUE), indicates how to change an input, while keeping it on the data manifold.
arXiv Detail & Related papers (2020-06-11T21:53:15Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.