Generalized Gaussian Temporal Difference Error for Uncertainty-aware Reinforcement Learning

• URL: http://arxiv.org/abs/2408.02295v2
• Date: Wed, 2 Oct 2024 05:46:06 GMT
• Title: Generalized Gaussian Temporal Difference Error for Uncertainty-aware Reinforcement Learning
• Authors: Seyeon Kim, Joonhun Lee, Namhoon Cho, Sungjun Han, Wooseop Hwang,
• Abstract summary: We introduce a novel framework for generalized Gaussian error modeling in deep reinforcement learning. Our framework enhances the flexibility of error distribution modeling by incorporating additional higher-order moment, particularly kurtosis.
• Score: 0.19418036471925312
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Conventional uncertainty-aware temporal difference (TD) learning methods often rely on simplistic assumptions, typically including a zero-mean Gaussian distribution for TD errors. Such oversimplification can lead to inaccurate error representations and compromised uncertainty estimation. In this paper, we introduce a novel framework for generalized Gaussian error modeling in deep reinforcement learning, applicable to both discrete and continuous control settings. Our framework enhances the flexibility of error distribution modeling by incorporating additional higher-order moment, particularly kurtosis, thereby improving the estimation and mitigation of data-dependent noise, i.e., aleatoric uncertainty. We examine the influence of the shape parameter of the generalized Gaussian distribution (GGD) on aleatoric uncertainty and provide a closed-form expression that demonstrates an inverse relationship between uncertainty and the shape parameter. Additionally, we propose a theoretically grounded weighting scheme to fully leverage the GGD. To address epistemic uncertainty, we enhance the batch inverse variance weighting by incorporating bias reduction and kurtosis considerations, resulting in improved robustness. Extensive experimental evaluations using policy gradient algorithms demonstrate the consistent efficacy of our method, showcasing significant performance improvements.

Related papers

• Evaluation of uncertainty estimations for Gaussian process regression based machine learning interatomic potentials [0.0]
  Uncertainty estimations for machine learning interatomic potentials are crucial to quantify the additional model error they introduce. We consider GPR models with Coulomb and SOAP representations as inputs to predict potential energy surfaces and excitation energies of molecules. We evaluate, how the GPR variance and ensemble-based uncertainties relate to the error and whether model performance improves by selecting the most uncertain samples from a fixed configuration space.
  arXiv  Detail & Related papers  (2024-10-27T10:06:09Z)
• Revisiting Essential and Nonessential Settings of Evidential Deep Learning [70.82728812001807]
  Evidential Deep Learning (EDL) is an emerging method for uncertainty estimation. We propose Re-EDL, a simplified yet more effective variant of EDL.
  arXiv  Detail & Related papers  (2024-10-01T04:27:07Z)
• Selective Learning: Towards Robust Calibration with Dynamic Regularization [79.92633587914659]
  Miscalibration in deep learning refers to there is a discrepancy between the predicted confidence and performance. We introduce Dynamic Regularization (DReg) which aims to learn what should be learned during training thereby circumventing the confidence adjusting trade-off.
  arXiv  Detail & Related papers  (2024-02-13T11:25:20Z)
• Mitigating Covariate Shift in Misspecified Regression with Applications to Reinforcement Learning [39.02112341007981]
  We study the effect of distribution shift in the presence of model misspecification. We show that empirical risk minimization, or standard least squares regression, can result in undesirable misspecification amplification. We develop a new algorithm that avoids this undesirable behavior, resulting in no misspecification amplification while still obtaining optimal statistical rates.
  arXiv  Detail & Related papers  (2024-01-22T18:59:12Z)
• Model-Based Uncertainty in Value Functions [89.31922008981735]
  We focus on characterizing the variance over values induced by a distribution over MDPs. Previous work upper bounds the posterior variance over values by solving a so-called uncertainty Bellman equation. We propose a new uncertainty Bellman equation whose solution converges to the true posterior variance over values.
  arXiv  Detail & Related papers  (2023-02-24T09:18:27Z)
• Leveraging Heteroscedastic Uncertainty in Learning Complex Spectral Mapping for Single-channel Speech Enhancement [20.823177372464414]
  Most speech enhancement (SE) models learn a point estimate, and do not make use of uncertainty estimation in the learning process. We show that modeling heteroscedastic uncertainty by minimizing a multivariate Gaussian negative log-likelihood (NLL) improves SE performance at no extra cost.
  arXiv  Detail & Related papers  (2022-11-16T02:29:05Z)
• Robustness and Accuracy Could Be Reconcilable by (Proper) Definition [109.62614226793833]
  The trade-off between robustness and accuracy has been widely studied in the adversarial literature. We find that it may stem from the improperly defined robust error, which imposes an inductive bias of local invariance. By definition, SCORE facilitates the reconciliation between robustness and accuracy, while still handling the worst-case uncertainty.
  arXiv  Detail & Related papers  (2022-02-21T10:36:09Z)
• Aleatoric uncertainty for Errors-in-Variables models in deep regression [0.48733623015338234]
  We show how the concept of Errors-in-Variables can be used in Bayesian deep regression. We discuss the approach along various simulated and real examples.
  arXiv  Detail & Related papers  (2021-05-19T12:37:02Z)
• Unlabelled Data Improves Bayesian Uncertainty Calibration under Covariate Shift [100.52588638477862]
  We develop an approximate Bayesian inference scheme based on posterior regularisation. We demonstrate the utility of our method in the context of transferring prognostic models of prostate cancer across globally diverse populations.
  arXiv  Detail & Related papers  (2020-06-26T13:50:19Z)
• Learning to Predict Error for MRI Reconstruction [67.76632988696943]
  We demonstrate that predictive uncertainty estimated by the current methods does not highly correlate with prediction error. We propose a novel method that estimates the target labels and magnitude of the prediction error in two steps.
  arXiv  Detail & Related papers  (2020-02-13T15:55:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.