Deeply Uncertain: Comparing Methods of Uncertainty Quantification in Deep Learning Algorithms

• URL: http://arxiv.org/abs/2004.10710v3
• Date: Wed, 22 Jul 2020 17:21:41 GMT
• Title: Deeply Uncertain: Comparing Methods of Uncertainty Quantification in Deep Learning Algorithms
• Authors: Jo\~ao Caldeira, Brian Nord
• Abstract summary: Three of the most common uncertainty quantification methods are compared to the standard analytic error propagation. Our results highlight some pitfalls that may occur when using these UQ methods.
• Score: 2.635832975589208
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present a comparison of methods for uncertainty quantification (UQ) in deep learning algorithms in the context of a simple physical system. Three of the most common uncertainty quantification methods - Bayesian Neural Networks (BNN), Concrete Dropout (CD), and Deep Ensembles (DE) - are compared to the standard analytic error propagation. We discuss this comparison in terms endemic to both machine learning ("epistemic" and "aleatoric") and the physical sciences ("statistical" and "systematic"). The comparisons are presented in terms of simulated experimental measurements of a single pendulum - a prototypical physical system for studying measurement and analysis techniques. Our results highlight some pitfalls that may occur when using these UQ methods. For example, when the variation of noise in the training set is small, all methods predicted the same relative uncertainty independently of the inputs. This issue is particularly hard to avoid in BNN. On the other hand, when the test set contains samples far from the training distribution, we found that no methods sufficiently increased the uncertainties associated to their predictions. This problem was particularly clear for CD. In light of these results, we make some recommendations for usage and interpretation of UQ methods.

Related papers

• The Statistical Accuracy of Neural Posterior and Likelihood Estimation [0.5825410941577592]
  We give an in-depth exploration on the statistical behavior of NPE and NLE. We prove that these methods have similar theoretical guarantees to common statistical methods like approximate Bayesian computation (ABC) and Bayesian synthetic likelihood (BSL) We prove that this accuracy can often be achieved at a vastly reduced computational cost, and will therefore deliver more attractive approximations than ABC and BSL in certain problems.
  arXiv  Detail & Related papers  (2024-11-18T21:25:32Z)
• DeepUQ: Assessing the Aleatoric Uncertainties from two Deep Learning Methods [44.99833362998488]
  We systematically compare aleatoric uncertainty measured by two UQ techniques, Deep Ensembles (DE) and Deep Evidential Regression (DER) Our method focuses on both zero-dimensional (0D) and two-dimensional (2D) data, to explore how the UQ methods function for different data dimensionalities. The predicted uncertainty is the least accurate for both UQ methods for the 2D input uncertainty experiment and the high-noise level.
  arXiv  Detail & Related papers  (2024-11-13T13:11:49Z)
• A General Framework for Uncertainty Quantification via Neural SDE-RNN [0.3314882635954751]
  Uncertainty quantification is a critical yet unsolved challenge for deep learning. We propose a novel framework based on the principles of recurrent neural networks and neural differential equations for reconciling irregularly sampled measurements. Our experiments on the IEEE 37 bus test system reveal that our framework can outperform state-of-the-art uncertainty quantification approaches for time-series data imputations.
  arXiv  Detail & Related papers  (2023-06-01T22:59:45Z)
• Uncertainty Aware Neural Network from Similarity and Sensitivity [19.688986566942]
  We present a neural network training method that considers similar samples with sensitivity awareness. We construct initial uncertainty bounds (UB) by considering the distribution of sensitivity aware similar samples. As following all the steps for finding UB for each sample requires a lot of computation and memory access, we train a UB computation NN.
  arXiv  Detail & Related papers  (2023-04-27T02:05:31Z)
• What is Flagged in Uncertainty Quantification? Latent Density Models for Uncertainty Categorization [68.15353480798244]
  Uncertainty Quantification (UQ) is essential for creating trustworthy machine learning models. Recent years have seen a steep rise in UQ methods that can flag suspicious examples. We propose a framework for categorizing uncertain examples flagged by UQ methods in classification tasks.
  arXiv  Detail & Related papers  (2022-07-11T19:47:00Z)
• The Optimal Noise in Noise-Contrastive Learning Is Not What You Think [80.07065346699005]
  We show that deviating from this assumption can actually lead to better statistical estimators. In particular, the optimal noise distribution is different from the data's and even from a different family.
  arXiv  Detail & Related papers  (2022-03-02T13:59:20Z)
• NUQ: Nonparametric Uncertainty Quantification for Deterministic Neural Networks [151.03112356092575]
  We show the principled way to measure the uncertainty of predictions for a classifier based on Nadaraya-Watson's nonparametric estimate of the conditional label distribution. We demonstrate the strong performance of the method in uncertainty estimation tasks on a variety of real-world image datasets.
  arXiv  Detail & Related papers  (2022-02-07T12:30:45Z)
• Quantifying Uncertainty in Deep Spatiotemporal Forecasting [67.77102283276409]
  We describe two types of forecasting problems: regular grid-based and graph-based. We analyze UQ methods from both the Bayesian and the frequentist point view, casting in a unified framework via statistical decision theory. Through extensive experiments on real-world road network traffic, epidemics, and air quality forecasting tasks, we reveal the statistical computational trade-offs for different UQ methods.
  arXiv  Detail & Related papers  (2021-05-25T14:35:46Z)
• An Empirical Comparison of Instance Attribution Methods for NLP [62.63504976810927]
  We evaluate the degree to which different potential instance attribution agree with respect to the importance of training samples. We find that simple retrieval methods yield training instances that differ from those identified via gradient-based methods.
  arXiv  Detail & Related papers  (2021-04-09T01:03:17Z)
• Detangling robustness in high dimensions: composite versus model-averaged estimation [11.658462692891355]
  Robust methods, though ubiquitous in practice, are yet to be fully understood in the context of regularized estimation and high dimensions. This paper provides a toolbox to further study robustness in these settings and focuses on prediction.
  arXiv  Detail & Related papers  (2020-06-12T20:40:15Z)
• Pitfalls of In-Domain Uncertainty Estimation and Ensembling in Deep Learning [70.72363097550483]
  In this study, we focus on in-domain uncertainty for image classification. To provide more insight in this study, we introduce the deep ensemble equivalent score (DEE)
  arXiv  Detail & Related papers  (2020-02-15T23:28:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.