DeepUQ: Assessing the Aleatoric Uncertainties from two Deep Learning Methods

• URL: http://arxiv.org/abs/2411.08587v1
• Date: Wed, 13 Nov 2024 13:11:49 GMT
• Title: DeepUQ: Assessing the Aleatoric Uncertainties from two Deep Learning Methods
• Authors: Rebecca Nevin, Aleksandra Ćiprijanović, Brian D. Nord,
• Abstract summary: We systematically compare aleatoric uncertainty measured by two UQ techniques, Deep Ensembles (DE) and Deep Evidential Regression (DER) Our method focuses on both zero-dimensional (0D) and two-dimensional (2D) data, to explore how the UQ methods function for different data dimensionalities. The predicted uncertainty is the least accurate for both UQ methods for the 2D input uncertainty experiment and the high-noise level.
• Score: 44.99833362998488
• License:
• Abstract: Assessing the quality of aleatoric uncertainty estimates from uncertainty quantification (UQ) deep learning methods is important in scientific contexts, where uncertainty is physically meaningful and important to characterize and interpret exactly. We systematically compare aleatoric uncertainty measured by two UQ techniques, Deep Ensembles (DE) and Deep Evidential Regression (DER). Our method focuses on both zero-dimensional (0D) and two-dimensional (2D) data, to explore how the UQ methods function for different data dimensionalities. We investigate uncertainty injected on the input and output variables and include a method to propagate uncertainty in the case of input uncertainty so that we can compare the predicted aleatoric uncertainty to the known values. We experiment with three levels of noise. The aleatoric uncertainty predicted across all models and experiments scales with the injected noise level. However, the predicted uncertainty is miscalibrated to $\rm{std}(\sigma_{\rm al})$ with the true uncertainty for half of the DE experiments and almost all of the DER experiments. The predicted uncertainty is the least accurate for both UQ methods for the 2D input uncertainty experiment and the high-noise level. While these results do not apply to more complex data, they highlight that further research on post-facto calibration for these methods would be beneficial, particularly for high-noise and high-dimensional settings.

Related papers

• How disentangled are your classification uncertainties? [6.144680854063938]
  Uncertainty Quantification in Machine Learning has progressed to predicting the source of uncertainty in a prediction. This work proposes a set of experiments to evaluate disentanglement of aleatoric and epistemic uncertainty.
  arXiv  Detail & Related papers  (2024-08-22T07:42:43Z)
• Uncertainty Quantification for Forward and Inverse Problems of PDEs via Latent Global Evolution [110.99891169486366]
  We propose a method that integrates efficient and precise uncertainty quantification into a deep learning-based surrogate model. Our method endows deep learning-based surrogate models with robust and efficient uncertainty quantification capabilities for both forward and inverse problems. Our method excels at propagating uncertainty over extended auto-regressive rollouts, making it suitable for scenarios involving long-term predictions.
  arXiv  Detail & Related papers  (2024-02-13T11:22:59Z)
• Calibrated Uncertainty Quantification for Operator Learning via Conformal Prediction [95.75771195913046]
  We propose a risk-controlling quantile neural operator, a distribution-free, finite-sample functional calibration conformal prediction method. We provide a theoretical calibration guarantee on the coverage rate, defined as the expected percentage of points on the function domain. Empirical results on a 2D Darcy flow and a 3D car surface pressure prediction task validate our theoretical results.
  arXiv  Detail & Related papers  (2024-02-02T23:43:28Z)
• One step closer to unbiased aleatoric uncertainty estimation [71.55174353766289]
  We propose a new estimation method by actively de-noising the observed data. By conducting a broad range of experiments, we demonstrate that our proposed approach provides a much closer approximation to the actual data uncertainty than the standard method.
  arXiv  Detail & Related papers  (2023-12-16T14:59:11Z)
• Reliability-Aware Prediction via Uncertainty Learning for Person Image Retrieval [51.83967175585896]
  UAL aims at providing reliability-aware predictions by considering data uncertainty and model uncertainty simultaneously. Data uncertainty captures the noise" inherent in the sample, while model uncertainty depicts the model's confidence in the sample's prediction.
  arXiv  Detail & Related papers  (2022-10-24T17:53:20Z)
• A Deeper Look into Aleatoric and Epistemic Uncertainty Disentanglement [7.6146285961466]
  In this paper, we generalize methods to produce disentangled uncertainties to work with different uncertainty quantification methods. We show that there is an interaction between learning aleatoric and epistemic uncertainty, which is unexpected and violates assumptions on aleatoric uncertainty. We expect that our formulation and results help practitioners and researchers choose uncertainty methods and expand the use of disentangled uncertainties.
  arXiv  Detail & Related papers  (2022-04-20T08:41:37Z)
• How certain are your uncertainties? [0.3655021726150368]
  Measures of uncertainty in the output of a deep learning method are useful in several ways. This work investigates the stability of these uncertainty measurements, in terms of both magnitude and spatial pattern.
  arXiv  Detail & Related papers  (2022-03-01T05:25:02Z)
• Discriminative Jackknife: Quantifying Uncertainty in Deep Learning via Higher-Order Influence Functions [121.10450359856242]
  We develop a frequentist procedure that utilizes influence functions of a model's loss functional to construct a jackknife (or leave-one-out) estimator of predictive confidence intervals. The DJ satisfies (1) and (2), is applicable to a wide range of deep learning models, is easy to implement, and can be applied in a post-hoc fashion without interfering with model training or compromising its accuracy.
  arXiv  Detail & Related papers  (2020-06-29T13:36:52Z)
• Deeply Uncertain: Comparing Methods of Uncertainty Quantification in Deep Learning Algorithms [2.635832975589208]
  Three of the most common uncertainty quantification methods are compared to the standard analytic error propagation. Our results highlight some pitfalls that may occur when using these UQ methods.
  arXiv  Detail & Related papers  (2020-04-22T17:13:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.