Improving MC-Dropout Uncertainty Estimates with Calibration Error-based Optimization

• URL: http://arxiv.org/abs/2110.03260v1
• Date: Thu, 7 Oct 2021 08:31:23 GMT
• Title: Improving MC-Dropout Uncertainty Estimates with Calibration Error-based Optimization
• Authors: Afshar Shamsi, Hamzeh Asgharnezhad, Moloud Abdar, AmirReza Tajally, Abbas Khosravi, Saeid Nahavandi, and Henry Leung
• Abstract summary: We propose two new loss functions by combining cross entropy with Expected Error (ECE) and Predictive Entropy (PE) Our results confirmed the great impact of the new hybrid loss functions for minimising the overlap between the distributions of uncertainty estimates for correct and incorrect predictions without sacrificing the model's overall performance.
• Score: 18.22429945073576
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Uncertainty quantification of machine learning and deep learning methods plays an important role in enhancing trust to the obtained result. In recent years, a numerous number of uncertainty quantification methods have been introduced. Monte Carlo dropout (MC-Dropout) is one of the most well-known techniques to quantify uncertainty in deep learning methods. In this study, we propose two new loss functions by combining cross entropy with Expected Calibration Error (ECE) and Predictive Entropy (PE). The obtained results clearly show that the new proposed loss functions lead to having a calibrated MC-Dropout method. Our results confirmed the great impact of the new hybrid loss functions for minimising the overlap between the distributions of uncertainty estimates for correct and incorrect predictions without sacrificing the model's overall performance.

Related papers

• Uncertainty-Calibrated Test-Time Model Adaptation without Forgetting [55.17761802332469]
  Test-time adaptation (TTA) seeks to tackle potential distribution shifts between training and test data by adapting a given model w.r.t. any test sample. Prior methods perform backpropagation for each test sample, resulting in unbearable optimization costs to many applications. We propose an Efficient Anti-Forgetting Test-Time Adaptation (EATA) method which develops an active sample selection criterion to identify reliable and non-redundant samples.
  arXiv  Detail & Related papers  (2024-03-18T05:49:45Z)
• Uncertainty Estimation by Fisher Information-based Evidential Deep Learning [61.94125052118442]
  Uncertainty estimation is a key factor that makes deep learning reliable in practical applications. We propose a novel method, Fisher Information-based Evidential Deep Learning ($mathcalI$-EDL) In particular, we introduce Fisher Information Matrix (FIM) to measure the informativeness of evidence carried by each sample, according to which we can dynamically reweight the objective loss terms to make the network more focused on the representation learning of uncertain classes.
  arXiv  Detail & Related papers  (2023-03-03T16:12:59Z)
• Model-Based Uncertainty in Value Functions [89.31922008981735]
  We focus on characterizing the variance over values induced by a distribution over MDPs. Previous work upper bounds the posterior variance over values by solving a so-called uncertainty Bellman equation. We propose a new uncertainty Bellman equation whose solution converges to the true posterior variance over values.
  arXiv  Detail & Related papers  (2023-02-24T09:18:27Z)
• On Calibrated Model Uncertainty in Deep Learning [0.0]
  We extend the approximate inference for the loss-calibrated Bayesian framework to dropweights based Bayesian neural networks. We show that decisions informed by loss-calibrated uncertainty can improve diagnostic performance to a greater extent than straightforward alternatives.
  arXiv  Detail & Related papers  (2022-06-15T20:16:32Z)
• On the Difficulty of Epistemic Uncertainty Quantification in Machine Learning: The Case of Direct Uncertainty Estimation through Loss Minimisation [8.298716599039501]
  Uncertainty quantification has received increasing attention in machine learning. The latter refers to the learner's (lack of) knowledge and appears to be especially difficult to measure and quantify. We show that loss minimisation does not work for second-order predictors.
  arXiv  Detail & Related papers  (2022-03-11T17:26:05Z)
• Diffusion Tensor Estimation with Uncertainty Calibration [6.5085381751712506]
  We propose a deep learning method to estimate the diffusion tensor and compute the estimation uncertainty. Data-dependent uncertainty is computed directly by the network and learned via loss attenuation. We show that the estimation uncertainties computed by the new method can highlight the model's biases, detect domain shift, and reflect the strength of noise in the measurements.
  arXiv  Detail & Related papers  (2021-11-21T15:58:01Z)
• DEUP: Direct Epistemic Uncertainty Prediction [56.087230230128185]
  Epistemic uncertainty is part of out-of-sample prediction error due to the lack of knowledge of the learner. We propose a principled approach for directly estimating epistemic uncertainty by learning to predict generalization error and subtracting an estimate of aleatoric uncertainty.
  arXiv  Detail & Related papers  (2021-02-16T23:50:35Z)
• Unlabelled Data Improves Bayesian Uncertainty Calibration under Covariate Shift [100.52588638477862]
  We develop an approximate Bayesian inference scheme based on posterior regularisation. We demonstrate the utility of our method in the context of transferring prognostic models of prostate cancer across globally diverse populations.
  arXiv  Detail & Related papers  (2020-06-26T13:50:19Z)
• Calibration of Model Uncertainty for Dropout Variational Inference [1.8065361710947976]
  In this paper, different logit scaling methods are extended to dropout variational inference to recalibrate model uncertainty. Experimental results show that logit scaling considerably reduce miscalibration by means of UCE.
  arXiv  Detail & Related papers  (2020-06-20T14:12:55Z)
• Learning to Predict Error for MRI Reconstruction [67.76632988696943]
  We demonstrate that predictive uncertainty estimated by the current methods does not highly correlate with prediction error. We propose a novel method that estimates the target labels and magnitude of the prediction error in two steps.
  arXiv  Detail & Related papers  (2020-02-13T15:55:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.