Actor-Critic Algorithm for Dynamic Expectile and CVaR

• URL: http://arxiv.org/abs/2605.07857v1
• Date: Fri, 08 May 2026 15:19:11 GMT
• Title: Actor-Critic Algorithm for Dynamic Expectile and CVaR
• Abstract summary: We propose a surrogate policy gradient without transition perturbation under softmax policy parameterization.<n>We also develop model-free value learning methods for dynamic expectile and conditional value-at-risk.
• Score: 10.868006419885601
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Optimizing dynamic risk with stochastic policies is challenging in both policy updates and value learning. The former typically requires transition perturbation, while the latter may rely on model-based approaches. To address these challenges, we propose a surrogate policy gradient without transition perturbation under softmax policy parameterization. We further develop model-free value learning methods for dynamic expectile and conditional value-at-risk by leveraging elicitability. Finally, inspired by Expected SARSA and Expected Policy Gradient, a model-free off-policy actor-critic algorithm is constructed. Empirical results in domains with verifiable risk-averse behavior show that our algorithm can learn risk-averse policy and consistently outperforms other existing methods.

Related papers

• Risk-Sensitive Exponential Actor Critic [8.650002078377485]
  We show that risk-sensitive exponential actor-critic (rsEAC) produces more numerically stable updates compared to existing approaches.<n>rsEAC reliably learns risk-sensitive policies in challenging risky variants of continuous tasks in MuJoCo.
  arXiv  Detail & Related papers  (2026-02-06T21:23:43Z)
• Dense Policy: Bidirectional Autoregressive Learning of Actions [51.60428100831717]
  This paper introduces a bidirectionally expanded learning approach, termed Dense Policy, to establish a new paradigm for autoregressive policies in action prediction.<n>It employs a lightweight encoder-only architecture to iteratively unfold the action sequence from an initial single frame into the target sequence in a coarse-to-fine manner.<n>Experiments validate that our dense policy has superior autoregressive learning capabilities and can surpass existing holistic generative policies.
  arXiv  Detail & Related papers  (2025-03-17T14:28:08Z)
• Risk-averse Total-reward MDPs with ERM and EVaR [12.719528972742394]
  We show that the risk-averse em total reward criterion can be optimized by a stationary policy.<n>Our results indicate that the total reward criterion may be preferable to the discounted criterion in a broad range of risk-averse reinforcement learning domains.
  arXiv  Detail & Related papers  (2024-08-30T13:33:18Z)
• Model-Based Epistemic Variance of Values for Risk-Aware Policy Optimization [59.758009422067]
  We consider the problem of quantifying uncertainty over expected cumulative rewards in model-based reinforcement learning. We propose a new uncertainty Bellman equation (UBE) whose solution converges to the true posterior variance over values. We introduce a general-purpose policy optimization algorithm, Q-Uncertainty Soft Actor-Critic (QU-SAC) that can be applied for either risk-seeking or risk-averse policy optimization.
  arXiv  Detail & Related papers  (2023-12-07T15:55:58Z)
• Probabilistic Reach-Avoid for Bayesian Neural Networks [71.67052234622781]
  We show that an optimal synthesis algorithm can provide more than a four-fold increase in the number of certifiable states. The algorithm is able to provide more than a three-fold increase in the average guaranteed reach-avoid probability.
  arXiv  Detail & Related papers  (2023-10-03T10:52:21Z)
• Distributional Method for Risk Averse Reinforcement Learning [0.0]
  We introduce a distributional method for learning the optimal policy in risk averse Markov decision process. We assume sequential observations of states, actions, and costs and assess the performance of a policy using dynamic risk measures.
  arXiv  Detail & Related papers  (2023-02-27T19:48:42Z)
• Model-based Safe Deep Reinforcement Learning via a Constrained Proximal Policy Optimization Algorithm [4.128216503196621]
  We propose an On-policy Model-based Safe Deep RL algorithm in which we learn the transition dynamics of the environment in an online manner. We show that our algorithm is more sample efficient and results in lower cumulative hazard violations as compared to constrained model-free approaches.
  arXiv  Detail & Related papers  (2022-10-14T06:53:02Z)
• Reinforcement Learning with Dynamic Convex Risk Measures [0.0]
  We develop an approach for solving time-consistent risk-sensitive optimization problems using model-free reinforcement learning (RL) We employ a time-consistent dynamic programming principle to determine the value of a particular policy, and develop policy gradient update rules.
  arXiv  Detail & Related papers  (2021-12-26T16:41:05Z)
• Learning Robust Policy against Disturbance in Transition Dynamics via State-Conservative Policy Optimization [63.75188254377202]
  Deep reinforcement learning algorithms can perform poorly in real-world tasks due to discrepancy between source and target environments. We propose a novel model-free actor-critic algorithm to learn robust policies without modeling the disturbance in advance. Experiments in several robot control tasks demonstrate that SCPO learns robust policies against the disturbance in transition dynamics.
  arXiv  Detail & Related papers  (2021-12-20T13:13:05Z)
• Policy Gradient Bayesian Robust Optimization for Imitation Learning [49.881386773269746]
  We derive a novel policy gradient-style robust optimization approach, PG-BROIL, to balance expected performance and risk. Results suggest PG-BROIL can produce a family of behaviors ranging from risk-neutral to risk-averse.
  arXiv  Detail & Related papers  (2021-06-11T16:49:15Z)
• Risk-Sensitive Deep RL: Variance-Constrained Actor-Critic Provably Finds Globally Optimal Policy [95.98698822755227]
  We make the first attempt to study risk-sensitive deep reinforcement learning under the average reward setting with the variance risk criteria. We propose an actor-critic algorithm that iteratively and efficiently updates the policy, the Lagrange multiplier, and the Fenchel dual variable.
  arXiv  Detail & Related papers  (2020-12-28T05:02:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.