Improving Uncertainty Calibration via Prior Augmented Data

• URL: http://arxiv.org/abs/2102.10803v1
• Date: Mon, 22 Feb 2021 07:02:37 GMT
• Title: Improving Uncertainty Calibration via Prior Augmented Data
• Authors: Jeffrey Willette, Juho Lee, Sung Ju Hwang
• Abstract summary: Neural networks have proven successful at learning from complex data distributions by acting as universal function approximators. They are often overconfident in their predictions, which leads to inaccurate and miscalibrated probabilistic predictions. We propose a solution by seeking out regions of feature space where the model is unjustifiably overconfident, and conditionally raising the entropy of those predictions towards that of the prior distribution of the labels.
• Score: 56.88185136509654
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Neural networks have proven successful at learning from complex data distributions by acting as universal function approximators. However, they are often overconfident in their predictions, which leads to inaccurate and miscalibrated probabilistic predictions. The problem of overconfidence becomes especially apparent in cases where the test-time data distribution differs from that which was seen during training. We propose a solution to this problem by seeking out regions of feature space where the model is unjustifiably overconfident, and conditionally raising the entropy of those predictions towards that of the prior distribution of the labels. Our method results in a better calibrated network and is agnostic to the underlying model structure, so it can be applied to any neural network which produces a probability density as an output. We demonstrate the effectiveness of our method and validate its performance on both classification and regression problems, applying it to recent probabilistic neural network models.

Related papers

• Tractable Function-Space Variational Inference in Bayesian Neural Networks [72.97620734290139]
  A popular approach for estimating the predictive uncertainty of neural networks is to define a prior distribution over the network parameters. We propose a scalable function-space variational inference method that allows incorporating prior information. We show that the proposed method leads to state-of-the-art uncertainty estimation and predictive performance on a range of prediction tasks.
  arXiv  Detail & Related papers  (2023-12-28T18:33:26Z)
• Uncertainty Quantification in Deep Neural Networks through Statistical Inference on Latent Space [0.0]
  We develop an algorithm that exploits the latent-space representation of data points fed into the network to assess the accuracy of their prediction. We show on a synthetic dataset that commonly used methods are mostly overconfident. In contrast, our method can detect such out-of-distribution data points as inaccurately predicted, thus aiding in the automatic detection of outliers.
  arXiv  Detail & Related papers  (2023-05-18T09:52:06Z)
• Confidence estimation of classification based on the distribution of the neural network output layer [4.529188601556233]
  One of the most common problems preventing the application of prediction models in the real world is lack of generalization. We propose novel methods that estimate uncertainty of particular predictions generated by a neural network classification model. The proposed methods infer the confidence of a particular prediction based on the distribution of the logit values corresponding to this prediction.
  arXiv  Detail & Related papers  (2022-10-14T12:32:50Z)
• Bayesian Neural Network Versus Ex-Post Calibration For Prediction Uncertainty [0.2343856409260935]
  Probabilistic predictions from neural networks account for predictive uncertainty during classification. In practice most datasets are trained on non-probabilistic neural networks which by default do not capture this inherent uncertainty. A plausible alternative to the calibration approach is to use Bayesian neural networks, which directly models a predictive distribution.
  arXiv  Detail & Related papers  (2022-09-29T07:22:19Z)
• Uncertainty Modeling for Out-of-Distribution Generalization [56.957731893992495]
  We argue that the feature statistics can be properly manipulated to improve the generalization ability of deep learning models. Common methods often consider the feature statistics as deterministic values measured from the learned features. We improve the network generalization ability by modeling the uncertainty of domain shifts with synthesized feature statistics during training.
  arXiv  Detail & Related papers  (2022-02-08T16:09:12Z)
• Conformal prediction for the design problem [72.14982816083297]
  In many real-world deployments of machine learning, we use a prediction algorithm to choose what data to test next. In such settings, there is a distinct type of distribution shift between the training and test data. We introduce a method to quantify predictive uncertainty in such settings.
  arXiv  Detail & Related papers  (2022-02-08T02:59:12Z)
• Dense Uncertainty Estimation [62.23555922631451]
  In this paper, we investigate neural networks and uncertainty estimation techniques to achieve both accurate deterministic prediction and reliable uncertainty estimation. We work on two types of uncertainty estimations solutions, namely ensemble based methods and generative model based methods, and explain their pros and cons while using them in fully/semi/weakly-supervised framework.
  arXiv  Detail & Related papers  (2021-10-13T01:23:48Z)
• Unlabelled Data Improves Bayesian Uncertainty Calibration under Covariate Shift [100.52588638477862]
  We develop an approximate Bayesian inference scheme based on posterior regularisation. We demonstrate the utility of our method in the context of transferring prognostic models of prostate cancer across globally diverse populations.
  arXiv  Detail & Related papers  (2020-06-26T13:50:19Z)
• A comprehensive study on the prediction reliability of graph neural networks for virtual screening [0.0]
  We investigate the effects of model architectures, regularization methods, and loss functions on the prediction performance and reliability of classification results. Our result highlights that correct choice of regularization and inference methods is evidently important to achieve high success rate.
  arXiv  Detail & Related papers  (2020-03-17T10:13:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.