Quantifying Aleatoric and Epistemic Uncertainty in Machine Learning: Are Conditional Entropy and Mutual Information Appropriate Measures?

• URL: http://arxiv.org/abs/2209.03302v2
• Date: Sun, 25 Jun 2023 09:49:38 GMT
• Title: Quantifying Aleatoric and Epistemic Uncertainty in Machine Learning: Are Conditional Entropy and Mutual Information Appropriate Measures?
• Authors: Lisa Wimmer and Yusuf Sale and Paul Hofman and Bern Bischl and Eyke H\"ullermeier
• Abstract summary: quantify aleatoric and epistemic uncertainty in terms of conditional entropy and mutual information. We identify various incoherencies that call their appropriateness into question. Experiments across different computer vision tasks support our theoretical findings.
• Score: 2.1655448059430222
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The quantification of aleatoric and epistemic uncertainty in terms of conditional entropy and mutual information, respectively, has recently become quite common in machine learning. While the properties of these measures, which are rooted in information theory, seem appealing at first glance, we identify various incoherencies that call their appropriateness into question. In addition to the measures themselves, we critically discuss the idea of an additive decomposition of total uncertainty into its aleatoric and epistemic constituents. Experiments across different computer vision tasks support our theoretical findings and raise concerns about current practice in uncertainty quantification.

Related papers

• Certainly Uncertain: A Benchmark and Metric for Multimodal Epistemic and Aleatoric Awareness [106.52630978891054]
  We present a taxonomy of uncertainty specific to vision-language AI systems. We also introduce a new metric confidence-weighted accuracy, that is well correlated with both accuracy and calibration error.
  arXiv  Detail & Related papers  (2024-07-02T04:23:54Z)
• To Believe or Not to Believe Your LLM [51.2579827761899]
  We explore uncertainty quantification in large language models (LLMs) We derive an information-theoretic metric that allows to reliably detect when only epistemic uncertainty is large. We conduct a series of experiments which demonstrate the advantage of our formulation.
  arXiv  Detail & Related papers  (2024-06-04T17:58:18Z)
• Is the Volume of a Credal Set a Good Measure for Epistemic Uncertainty? [2.658812114255374]
  Uncertainty measures have become imperative in various disciplines, especially in machine learning and intelligence. As an alternative to a measure of uncertainty, we consider credal sets. We show that a credal set is a meaningful measure of uncertainty in the case of classification but less so for multi-class classification.
  arXiv  Detail & Related papers  (2023-06-16T02:17:45Z)
• Sources of Uncertainty in Machine Learning -- A Statisticians' View [3.1498833540989413]
  The paper aims to formalize the two types of uncertainty associated with machine learning. Drawing parallels between statistical concepts and uncertainty in machine learning, we also demonstrate the role of data and their influence on uncertainty.
  arXiv  Detail & Related papers  (2023-05-26T07:44:19Z)
• The Unreasonable Effectiveness of Deep Evidential Regression [72.30888739450343]
  A new approach with uncertainty-aware regression-based neural networks (NNs) shows promise over traditional deterministic methods and typical Bayesian NNs. We detail the theoretical shortcomings and analyze the performance on synthetic and real-world data sets, showing that Deep Evidential Regression is a quantification rather than an exact uncertainty.
  arXiv  Detail & Related papers  (2022-05-20T10:10:32Z)
• Disturbance Enhanced Uncertainty Relations [2.075038010521211]
  We show that disturbance of one measurement to a subsequent measurement sets lower-bounds to uncertainty. The obtained relations, referred to as disturbance enhanced uncertainty relations, immediately find various applications in the field of quantum information. We anticipate that this new twist on uncertainty principle may shed new light on quantum foundations and may also inspire further applications in the field of quantum information.
  arXiv  Detail & Related papers  (2022-02-15T08:37:24Z)
• Ensemble-based Uncertainty Quantification: Bayesian versus Credal Inference [0.0]
  We consider ensemble-based approaches to uncertainty quantification. We specifically focus on Bayesian methods and approaches based on so-called credal sets. The effectiveness of corresponding measures is evaluated and compared in an empirical study on classification with a reject option.
  arXiv  Detail & Related papers  (2021-07-21T22:47:24Z)
• BayesIMP: Uncertainty Quantification for Causal Data Fusion [52.184885680729224]
  We study the causal data fusion problem, where datasets pertaining to multiple causal graphs are combined to estimate the average treatment effect of a target variable. We introduce a framework which combines ideas from probabilistic integration and kernel mean embeddings to represent interventional distributions in the reproducing kernel Hilbert space.
  arXiv  Detail & Related papers  (2021-06-07T10:14:18Z)
• DEUP: Direct Epistemic Uncertainty Prediction [56.087230230128185]
  Epistemic uncertainty is part of out-of-sample prediction error due to the lack of knowledge of the learner. We propose a principled approach for directly estimating epistemic uncertainty by learning to predict generalization error and subtracting an estimate of aleatoric uncertainty.
  arXiv  Detail & Related papers  (2021-02-16T23:50:35Z)
• The Hidden Uncertainty in a Neural Networks Activations [105.4223982696279]
  The distribution of a neural network's latent representations has been successfully used to detect out-of-distribution (OOD) data. This work investigates whether this distribution correlates with a model's epistemic uncertainty, thus indicating its ability to generalise to novel inputs.
  arXiv  Detail & Related papers  (2020-12-05T17:30:35Z)
• Uncertainty Quantification in Extreme Learning Machine: Analytical Developments, Variance Estimates and Confidence Intervals [0.0]
  Uncertainty quantification is crucial to assess prediction quality of a machine learning model. Most methods proposed in the literature make strong assumptions on the data, ignore the randomness of input weights or neglect the bias contribution in confidence interval estimations. This paper presents novel estimations that overcome these constraints and improve the understanding of ELM variability.
  arXiv  Detail & Related papers  (2020-11-03T13:45:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.