Sources of Uncertainty in Machine Learning -- A Statisticians' View
- URL: http://arxiv.org/abs/2305.16703v1
- Date: Fri, 26 May 2023 07:44:19 GMT
- Title: Sources of Uncertainty in Machine Learning -- A Statisticians' View
- Authors: Cornelia Gruber, Patrick Oliver Schenk, Malte Schierholz, Frauke
Kreuter, G\"oran Kauermann
- Abstract summary: The paper aims to formalize the two types of uncertainty associated with machine learning.
Drawing parallels between statistical concepts and uncertainty in machine learning, we also demonstrate the role of data and their influence on uncertainty.
- Score: 3.1498833540989413
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Machine Learning and Deep Learning have achieved an impressive standard
today, enabling us to answer questions that were inconceivable a few years ago.
Besides these successes, it becomes clear, that beyond pure prediction, which
is the primary strength of most supervised machine learning algorithms, the
quantification of uncertainty is relevant and necessary as well. While first
concepts and ideas in this direction have emerged in recent years, this paper
adopts a conceptual perspective and examines possible sources of uncertainty.
By adopting the viewpoint of a statistician, we discuss the concepts of
aleatoric and epistemic uncertainty, which are more commonly associated with
machine learning. The paper aims to formalize the two types of uncertainty and
demonstrates that sources of uncertainty are miscellaneous and can not always
be decomposed into aleatoric and epistemic. Drawing parallels between
statistical concepts and uncertainty in machine learning, we also demonstrate
the role of data and their influence on uncertainty.
Related papers
- Uncertainties of Latent Representations in Computer Vision [2.33877878310217]
This thesis makes uncertainty estimates easily accessible by adding them to the latent representation vectors of pretrained computer vision models.
We show that these unobservable uncertainties about unobservable latent representations are indeed provably correct.
arXiv Detail & Related papers (2024-08-26T14:02:30Z) - One step closer to unbiased aleatoric uncertainty estimation [71.55174353766289]
We propose a new estimation method by actively de-noising the observed data.
By conducting a broad range of experiments, we demonstrate that our proposed approach provides a much closer approximation to the actual data uncertainty than the standard method.
arXiv Detail & Related papers (2023-12-16T14:59:11Z) - Evidential Deep Learning: Enhancing Predictive Uncertainty Estimation
for Earth System Science Applications [0.32302664881848275]
Evidential deep learning is a technique that extends parametric deep learning to higher-order distributions.
This study compares the uncertainty derived from evidential neural networks to those obtained from ensembles.
We show evidential deep learning models attaining predictive accuracy rivaling standard methods, while robustly quantifying both sources of uncertainty.
arXiv Detail & Related papers (2023-09-22T23:04:51Z) - Uncertainty in Natural Language Processing: Sources, Quantification, and
Applications [56.130945359053776]
We provide a comprehensive review of uncertainty-relevant works in the NLP field.
We first categorize the sources of uncertainty in natural language into three types, including input, system, and output.
We discuss the challenges of uncertainty estimation in NLP and discuss potential future directions.
arXiv Detail & Related papers (2023-06-05T06:46:53Z) - On Second-Order Scoring Rules for Epistemic Uncertainty Quantification [8.298716599039501]
We show that there seems to be no loss function that provides an incentive for a second-order learner to faithfully represent its uncertainty.
As a main mathematical tool to prove this result, we introduce the generalised notion of second-order scoring rules.
arXiv Detail & Related papers (2023-01-30T08:59:45Z) - The Unreasonable Effectiveness of Deep Evidential Regression [72.30888739450343]
A new approach with uncertainty-aware regression-based neural networks (NNs) shows promise over traditional deterministic methods and typical Bayesian NNs.
We detail the theoretical shortcomings and analyze the performance on synthetic and real-world data sets, showing that Deep Evidential Regression is a quantification rather than an exact uncertainty.
arXiv Detail & Related papers (2022-05-20T10:10:32Z) - Teaching Uncertainty Quantification in Machine Learning through Use
Cases [0.0]
Uncertainty in machine learning is not generally taught as general knowledge in Machine Learning course curricula.
We propose a short curriculum for a course about uncertainty in machine learning, and complement the course with a selection of use cases.
arXiv Detail & Related papers (2021-08-19T14:22:17Z) - Multi Agent System for Machine Learning Under Uncertainty in Cyber
Physical Manufacturing System [78.60415450507706]
Recent advancements in predictive machine learning has led to its application in various use cases in manufacturing.
Most research focused on maximising predictive accuracy without addressing the uncertainty associated with it.
In this paper, we determine the sources of uncertainty in machine learning and establish the success criteria of a machine learning system to function well under uncertainty.
arXiv Detail & Related papers (2021-07-28T10:28:05Z) - DEUP: Direct Epistemic Uncertainty Prediction [56.087230230128185]
Epistemic uncertainty is part of out-of-sample prediction error due to the lack of knowledge of the learner.
We propose a principled approach for directly estimating epistemic uncertainty by learning to predict generalization error and subtracting an estimate of aleatoric uncertainty.
arXiv Detail & Related papers (2021-02-16T23:50:35Z) - Uncertainty as a Form of Transparency: Measuring, Communicating, and
Using Uncertainty [66.17147341354577]
We argue for considering a complementary form of transparency by estimating and communicating the uncertainty associated with model predictions.
We describe how uncertainty can be used to mitigate model unfairness, augment decision-making, and build trustworthy systems.
This work constitutes an interdisciplinary review drawn from literature spanning machine learning, visualization/HCI, design, decision-making, and fairness.
arXiv Detail & Related papers (2020-11-15T17:26:14Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.