A unified uncertainty-aware exploration: Combining epistemic and aleatory uncertainty

• URL: http://arxiv.org/abs/2401.02914v1
• Date: Fri, 5 Jan 2024 17:39:00 GMT
• Title: A unified uncertainty-aware exploration: Combining epistemic and aleatory uncertainty
• Authors: Parvin Malekzadeh, Ming Hou, Konstantinos N. Plataniotis
• Abstract summary: We propose an algorithm that quantifies the combined effect of aleatory and epistemic uncertainty for risk-sensitive exploration. Our method builds on a novel extension of distributional RL that estimates a parameterized return distribution. Experimental results on tasks with exploration and risk challenges show that our method outperforms alternative approaches.
• Score: 21.139502047972684
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Exploration is a significant challenge in practical reinforcement learning (RL), and uncertainty-aware exploration that incorporates the quantification of epistemic and aleatory uncertainty has been recognized as an effective exploration strategy. However, capturing the combined effect of aleatory and epistemic uncertainty for decision-making is difficult. Existing works estimate aleatory and epistemic uncertainty separately and consider the composite uncertainty as an additive combination of the two. Nevertheless, the additive formulation leads to excessive risk-taking behavior, causing instability. In this paper, we propose an algorithm that clarifies the theoretical connection between aleatory and epistemic uncertainty, unifies aleatory and epistemic uncertainty estimation, and quantifies the combined effect of both uncertainties for a risk-sensitive exploration. Our method builds on a novel extension of distributional RL that estimates a parameterized return distribution whose parameters are random variables encoding epistemic uncertainty. Experimental results on tasks with exploration and risk challenges show that our method outperforms alternative approaches.

Related papers

• Predictive Uncertainty Quantification via Risk Decompositions for Strictly Proper Scoring Rules [7.0549244915538765]
  Uncertainty in predictive modeling often relies on ad hoc methods. This paper introduces a theoretical approach to understanding uncertainty through statistical risks. We show how to split pointwise risk into Bayes risk and excess risk.
  arXiv  Detail & Related papers  (2024-02-16T14:40:22Z)
• Is Epistemic Uncertainty Faithfully Represented by Evidential Deep Learning Methods? [26.344949402398917]
  This paper presents novel theoretical insights of evidential deep learning. It highlights the difficulties in optimizing second-order loss functions. It provides novel insights into issues of identifiability and convergence in second-order loss minimization.
  arXiv  Detail & Related papers  (2024-02-14T10:07:05Z)
• One step closer to unbiased aleatoric uncertainty estimation [71.55174353766289]
  We propose a new estimation method by actively de-noising the observed data. By conducting a broad range of experiments, we demonstrate that our proposed approach provides a much closer approximation to the actual data uncertainty than the standard method.
  arXiv  Detail & Related papers  (2023-12-16T14:59:11Z)
• Evidential uncertainty sampling for active learning [4.271684331748043]
  Recent studies in active learning have focused on the decomposition of model uncertainty into reducible and irreducible uncertainties. In this paper, the aim is to simplify the computational process while eliminating the dependence on observations.
  arXiv  Detail & Related papers  (2023-09-21T21:26:50Z)
• The Unreasonable Effectiveness of Deep Evidential Regression [72.30888739450343]
  A new approach with uncertainty-aware regression-based neural networks (NNs) shows promise over traditional deterministic methods and typical Bayesian NNs. We detail the theoretical shortcomings and analyze the performance on synthetic and real-world data sets, showing that Deep Evidential Regression is a quantification rather than an exact uncertainty.
  arXiv  Detail & Related papers  (2022-05-20T10:10:32Z)
• A Deeper Look into Aleatoric and Epistemic Uncertainty Disentanglement [7.6146285961466]
  In this paper, we generalize methods to produce disentangled uncertainties to work with different uncertainty quantification methods. We show that there is an interaction between learning aleatoric and epistemic uncertainty, which is unexpected and violates assumptions on aleatoric uncertainty. We expect that our formulation and results help practitioners and researchers choose uncertainty methods and expand the use of disentangled uncertainties.
  arXiv  Detail & Related papers  (2022-04-20T08:41:37Z)
• Dense Uncertainty Estimation via an Ensemble-based Conditional Latent Variable Model [68.34559610536614]
  We argue that the aleatoric uncertainty is an inherent attribute of the data and can only be correctly estimated with an unbiased oracle model. We propose a new sampling and selection strategy at train time to approximate the oracle model for aleatoric uncertainty estimation. Our results show that our solution achieves both accurate deterministic results and reliable uncertainty estimation.
  arXiv  Detail & Related papers  (2021-11-22T08:54:10Z)
• Enabling risk-aware Reinforcement Learning for medical interventions through uncertainty decomposition [9.208828373290487]
  Reinforcement Learning (RL) is emerging as tool for tackling complex control and decision-making problems. It is often challenging to bridge the gap between an apparently optimal policy learnt by an agent and its real-world deployment. Here we propose how a distributional approach (UA-DQN) can be recast to render uncertainties by decomposing the net effects of each uncertainty.
  arXiv  Detail & Related papers  (2021-09-16T09:36:53Z)
• DEUP: Direct Epistemic Uncertainty Prediction [56.087230230128185]
  Epistemic uncertainty is part of out-of-sample prediction error due to the lack of knowledge of the learner. We propose a principled approach for directly estimating epistemic uncertainty by learning to predict generalization error and subtracting an estimate of aleatoric uncertainty.
  arXiv  Detail & Related papers  (2021-02-16T23:50:35Z)
• The Hidden Uncertainty in a Neural Networks Activations [105.4223982696279]
  The distribution of a neural network's latent representations has been successfully used to detect out-of-distribution (OOD) data. This work investigates whether this distribution correlates with a model's epistemic uncertainty, thus indicating its ability to generalise to novel inputs.
  arXiv  Detail & Related papers  (2020-12-05T17:30:35Z)
• Temporal Difference Uncertainties as a Signal for Exploration [76.6341354269013]
  An effective approach to exploration in reinforcement learning is to rely on an agent's uncertainty over the optimal policy. In this paper, we highlight that value estimates are easily biased and temporally inconsistent. We propose a novel method for estimating uncertainty over the value function that relies on inducing a distribution over temporal difference errors.
  arXiv  Detail & Related papers  (2020-10-05T18:11:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.