Entropic Risk Constrained Soft-Robust Policy Optimization

• URL: http://arxiv.org/abs/2006.11679v1
• Date: Sat, 20 Jun 2020 23:48:28 GMT
• Title: Entropic Risk Constrained Soft-Robust Policy Optimization
• Authors: Reazul Hasan Russel, Bahram Behzadian, Marek Petrik
• Abstract summary: It is important in high-stakes domains to quantify and manage risk induced by model uncertainties. We propose an entropic risk constrained policy gradient and actor-critic algorithms that are risk-averse to the model uncertainty.
• Score: 12.362670630646805
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Having a perfect model to compute the optimal policy is often infeasible in reinforcement learning. It is important in high-stakes domains to quantify and manage risk induced by model uncertainties. Entropic risk measure is an exponential utility-based convex risk measure that satisfies many reasonable properties. In this paper, we propose an entropic risk constrained policy gradient and actor-critic algorithms that are risk-averse to the model uncertainty. We demonstrate the usefulness of our algorithms on several problem domains.

Related papers

• Robust Reinforcement Learning with Dynamic Distortion Risk Measures [0.0]
  We devise a framework to solve robust risk-aware reinforcement learning problems. We simultaneously account for environmental uncertainty and risk with a class of dynamic robust distortion risk measures. We construct an actor-critic algorithm to solve this class of robust risk-aware RL problems.
  arXiv  Detail & Related papers  (2024-09-16T08:54:59Z)
• Data-Adaptive Tradeoffs among Multiple Risks in Distribution-Free Prediction [55.77015419028725]
  We develop methods that permit valid control of risk when threshold and tradeoff parameters are chosen adaptively. Our methodology supports monotone and nearly-monotone risks, but otherwise makes no distributional assumptions.
  arXiv  Detail & Related papers  (2024-03-28T17:28:06Z)
• Model-Based Epistemic Variance of Values for Risk-Aware Policy Optimization [59.758009422067]
  We consider the problem of quantifying uncertainty over expected cumulative rewards in model-based reinforcement learning. We propose a new uncertainty Bellman equation (UBE) whose solution converges to the true posterior variance over values. We introduce a general-purpose policy optimization algorithm, Q-Uncertainty Soft Actor-Critic (QU-SAC) that can be applied for either risk-seeking or risk-averse policy optimization.
  arXiv  Detail & Related papers  (2023-12-07T15:55:58Z)
• Capsa: A Unified Framework for Quantifying Risk in Deep Neural Networks [142.67349734180445]
  Existing algorithms that provide risk-awareness to deep neural networks are complex and ad-hoc. Here we present capsa, a framework for extending models with risk-awareness.
  arXiv  Detail & Related papers  (2023-08-01T02:07:47Z)
• Regret Bounds for Risk-sensitive Reinforcement Learning with Lipschitz Dynamic Risk Measures [23.46659319363579]
  We present two model-based algorithms applied to emphLipschitz dynamic risk measures. Notably, our upper bounds demonstrate optimal dependencies on the number of actions and episodes.
  arXiv  Detail & Related papers  (2023-06-04T16:24:19Z)
• Efficient Risk-Averse Reinforcement Learning [79.61412643761034]
  In risk-averse reinforcement learning (RL), the goal is to optimize some risk measure of the returns. We prove that under certain conditions this inevitably leads to a local-optimum barrier, and propose a soft risk mechanism to bypass it. We demonstrate improved risk aversion in maze navigation, autonomous driving, and resource allocation benchmarks.
  arXiv  Detail & Related papers  (2022-05-10T19:40:52Z)
• A policy gradient approach for optimization of smooth risk measures [8.087699764574788]
  We consider episodic Markov decision processes, and model the risk using the broad class of smooth risk measures of the cumulative discounted reward. We propose two template policy gradient algorithms that optimize a smooth risk measure in on-policy and off-policy RL settings.
  arXiv  Detail & Related papers  (2022-02-22T17:26:28Z)
• Reinforcement Learning with Dynamic Convex Risk Measures [0.0]
  We develop an approach for solving time-consistent risk-sensitive optimization problems using model-free reinforcement learning (RL) We employ a time-consistent dynamic programming principle to determine the value of a particular policy, and develop policy gradient update rules.
  arXiv  Detail & Related papers  (2021-12-26T16:41:05Z)
• Risk Conditioned Neural Motion Planning [14.018786843419862]
  Risk-bounded motion planning is an important yet difficult problem for safety-critical tasks. We propose an extension of soft actor critic model to estimate the execution risk of a plan through a risk critic. We show the advantage of our model in terms of both computational time and plan quality, compared to a state-of-the-art mathematical programming baseline.
  arXiv  Detail & Related papers  (2021-08-04T05:33:52Z)
• Policy Gradient Bayesian Robust Optimization for Imitation Learning [49.881386773269746]
  We derive a novel policy gradient-style robust optimization approach, PG-BROIL, to balance expected performance and risk. Results suggest PG-BROIL can produce a family of behaviors ranging from risk-neutral to risk-averse.
  arXiv  Detail & Related papers  (2021-06-11T16:49:15Z)
• Risk-Sensitive Deep RL: Variance-Constrained Actor-Critic Provably Finds Globally Optimal Policy [95.98698822755227]
  We make the first attempt to study risk-sensitive deep reinforcement learning under the average reward setting with the variance risk criteria. We propose an actor-critic algorithm that iteratively and efficiently updates the policy, the Lagrange multiplier, and the Fenchel dual variable.
  arXiv  Detail & Related papers  (2020-12-28T05:02:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.