Enabling risk-aware Reinforcement Learning for medical interventions through uncertainty decomposition

• URL: http://arxiv.org/abs/2109.07827v1
• Date: Thu, 16 Sep 2021 09:36:53 GMT
• Title: Enabling risk-aware Reinforcement Learning for medical interventions through uncertainty decomposition
• Authors: Paul Festor, Giulia Luise, Matthieu Komorowski and A. Aldo Faisal
• Abstract summary: Reinforcement Learning (RL) is emerging as tool for tackling complex control and decision-making problems. It is often challenging to bridge the gap between an apparently optimal policy learnt by an agent and its real-world deployment. Here we propose how a distributional approach (UA-DQN) can be recast to render uncertainties by decomposing the net effects of each uncertainty.
• Score: 9.208828373290487
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reinforcement Learning (RL) is emerging as tool for tackling complex control and decision-making problems. However, in high-risk environments such as healthcare, manufacturing, automotive or aerospace, it is often challenging to bridge the gap between an apparently optimal policy learnt by an agent and its real-world deployment, due to the uncertainties and risk associated with it. Broadly speaking RL agents face two kinds of uncertainty, 1. aleatoric uncertainty, which reflects randomness or noise in the dynamics of the world, and 2. epistemic uncertainty, which reflects the bounded knowledge of the agent due to model limitations and finite amount of information/data the agent has acquired about the world. These two types of uncertainty carry fundamentally different implications for the evaluation of performance and the level of risk or trust. Yet these aleatoric and epistemic uncertainties are generally confounded as standard and even distributional RL is agnostic to this difference. Here we propose how a distributional approach (UA-DQN) can be recast to render uncertainties by decomposing the net effects of each uncertainty. We demonstrate the operation of this method in grid world examples to build intuition and then show a proof of concept application for an RL agent operating as a clinical decision support system in critical care

Related papers

• Know Where You're Uncertain When Planning with Multimodal Foundation Models: A Formal Framework [54.40508478482667]
  We present a comprehensive framework to disentangle, quantify, and mitigate uncertainty in perception and plan generation. We propose methods tailored to the unique properties of perception and decision-making. We show that our uncertainty disentanglement framework reduces variability by up to 40% and enhances task success rates by 5% compared to baselines.
  arXiv  Detail & Related papers  (2024-11-03T17:32:00Z)
• Controlling Risk of Retrieval-augmented Generation: A Counterfactual Prompting Framework [77.45983464131977]
  We focus on how likely it is that a RAG model's prediction is incorrect, resulting in uncontrollable risks in real-world applications. Our research identifies two critical latent factors affecting RAG's confidence in its predictions. We develop a counterfactual prompting framework that induces the models to alter these factors and analyzes the effect on their answers.
  arXiv  Detail & Related papers  (2024-09-24T14:52:14Z)
• Data-Adaptive Tradeoffs among Multiple Risks in Distribution-Free Prediction [55.77015419028725]
  We develop methods that permit valid control of risk when threshold and tradeoff parameters are chosen adaptively. Our methodology supports monotone and nearly-monotone risks, but otherwise makes no distributional assumptions.
  arXiv  Detail & Related papers  (2024-03-28T17:28:06Z)
• Auditing Fairness under Unobserved Confounding [56.61738581796362]
  We show that we can still give meaningful bounds on treatment rates to high-risk individuals, even when entirely eliminating or relaxing the assumption that all relevant risk factors are observed. This result is of immediate practical interest: we can audit unfair outcomes of existing decision-making systems in a principled manner.
  arXiv  Detail & Related papers  (2024-03-18T21:09:06Z)
• A unified uncertainty-aware exploration: Combining epistemic and aleatory uncertainty [21.139502047972684]
  We propose an algorithm that quantifies the combined effect of aleatory and epistemic uncertainty for risk-sensitive exploration. Our method builds on a novel extension of distributional RL that estimates a parameterized return distribution. Experimental results on tasks with exploration and risk challenges show that our method outperforms alternative approaches.
  arXiv  Detail & Related papers  (2024-01-05T17:39:00Z)
• Capsa: A Unified Framework for Quantifying Risk in Deep Neural Networks [142.67349734180445]
  Existing algorithms that provide risk-awareness to deep neural networks are complex and ad-hoc. Here we present capsa, a framework for extending models with risk-awareness.
  arXiv  Detail & Related papers  (2023-08-01T02:07:47Z)
• Seeing is not Believing: Robust Reinforcement Learning against Spurious Correlation [57.351098530477124]
  We consider one critical type of robustness against spurious correlation, where different portions of the state do not have correlations induced by unobserved confounders. A model that learns such useless or even harmful correlation could catastrophically fail when the confounder in the test case deviates from the training one. Existing robust algorithms that assume simple and unstructured uncertainty sets are therefore inadequate to address this challenge.
  arXiv  Detail & Related papers  (2023-07-15T23:53:37Z)
• Risk-Sensitive and Robust Model-Based Reinforcement Learning and Planning [2.627046865670577]
  We will address both planning and reinforcement learning approaches to sequential decision-making. In many real-world domains, it is impossible to construct a perfectly accurate model or simulator. We make a number of contributions towards this goal, with a focus on model-based algorithms.
  arXiv  Detail & Related papers  (2023-04-02T16:44:14Z)
• One Risk to Rule Them All: A Risk-Sensitive Perspective on Model-Based Offline Reinforcement Learning [25.218430053391884]
  We propose risk-sensitivity as a mechanism to jointly address both of these issues. Risk-aversion to aleatoric uncertainty discourages actions that may result in poor outcomes due to environmentity. Our experiments show that our algorithm achieves competitive performance on deterministic benchmarks.
  arXiv  Detail & Related papers  (2022-11-30T21:24:11Z)
• Disentangling Epistemic and Aleatoric Uncertainty in Reinforcement Learning [35.791555387656956]
  Aleatoric uncertainty results from the irreducible environment leading to inherently risky states and actions. Epistemic uncertainty results from the limited information accumulated during learning to make informed decisions. Characterizing aleatoric and uncertainty can be used to speed up learning in a training environment, improve generalization to similar testing environments, and flag unfamiliar behavior in anomalous testing environments.
  arXiv  Detail & Related papers  (2022-06-03T13:20:16Z)
• Ensemble Quantile Networks: Uncertainty-Aware Reinforcement Learning with Applications in Autonomous Driving [1.6758573326215689]
  Reinforcement learning can be used to create a decision-making agent for autonomous driving. Previous approaches provide only black-box solutions, which do not offer information on how confident the agent is about its decisions. This paper introduces the Ensemble Quantile Networks (EQN) method, which combines distributional RL with an ensemble approach to obtain a complete uncertainty estimate.
  arXiv  Detail & Related papers  (2021-05-21T10:36:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.