Risk-averse learning with delayed feedback

• URL: http://arxiv.org/abs/2409.16866v1
• Date: Wed, 25 Sep 2024 12:32:22 GMT
• Title: Risk-averse learning with delayed feedback
• Authors: Siyi Wang, Zifan Wang, Karl Henrik Johansson, Sandra Hirche,
• Abstract summary: We develop two risk-averse learning algorithms that rely on one-point and two-point zeroth-order optimization approaches. The results suggest that the two-point risk-averse learning achieves a smaller regret bound than the one-point algorithm.
• Score: 17.626195546400247
• License: http://creativecommons.org/publicdomain/zero/1.0/
• Abstract: In real-world scenarios, the impacts of decisions may not manifest immediately. Taking these delays into account facilitates accurate assessment and management of risk in real-world environments, thereby ensuring the efficacy of strategies. In this paper, we investigate risk-averse learning using Conditional Value at Risk (CVaR) as risk measure, while incorporating delayed feedback with unknown but bounded delays. We develop two risk-averse learning algorithms that rely on one-point and two-point zeroth-order optimization approaches, respectively. The regret achieved by the algorithms is analyzed in terms of the cumulative delay and the number of total samplings. The results suggest that the two-point risk-averse learning achieves a smaller regret bound than the one-point algorithm. Furthermore, the one-point risk-averse learning algorithm attains sublinear regret under certain delay conditions, and the two-point risk-averse learning algorithm achieves sublinear regret with minimal restrictions on the delay. We provide numerical experiments on a dynamic pricing problem to demonstrate the performance of the proposed algorithms.

Related papers

• Pessimism Meets Risk: Risk-Sensitive Offline Reinforcement Learning [19.292214425524303]
  We study risk-sensitive reinforcement learning (RL), a crucial field due to its ability to enhance decision-making in scenarios where it is essential to manage uncertainty and minimize potential adverse outcomes. Our work focuses on applying the entropic risk measure to RL problems. We center on the linear Markov Decision Process (MDP) setting, a well-regarded theoretical framework that has yet to be examined from a risk-sensitive standpoint.
  arXiv  Detail & Related papers  (2024-07-10T13:09:52Z)
• Risk-Aware Linear Bandits: Theory and Applications in Smart Order Routing [10.69955834942979]
  We consider risk-aware bandits optimization with applications in smart order routing (SOR) Driven by the variance-minimizing globally-optimal (G-optimal) design, we propose the novel instance-independent Risk-Aware Explore-then-Commit (RISE) algorithm and the instance-dependent Risk-Aware Successive Elimination (RISE++) algorithm.
  arXiv  Detail & Related papers  (2022-08-04T00:21:10Z)
• Deep Learning for Systemic Risk Measures [3.274367403737527]
  The aim of this paper is to study a new methodological framework for systemic risk measures. Under this new framework, systemic risk measures can be interpreted as the minimal amount of cash that secures the aggregated system. Deep learning is increasingly receiving attention in financial modelings and risk management.
  arXiv  Detail & Related papers  (2022-07-02T05:01:19Z)
• Conditionally Elicitable Dynamic Risk Measures for Deep Reinforcement Learning [0.0]
  We develop an efficient approach to estimate a class of dynamic spectral risk measures with deep neural networks. We also develop a risk-sensitive actor-critic algorithm that uses full episodes and does not require any additional nested transitions.
  arXiv  Detail & Related papers  (2022-06-29T14:11:15Z)
• Efficient Risk-Averse Reinforcement Learning [79.61412643761034]
  In risk-averse reinforcement learning (RL), the goal is to optimize some risk measure of the returns. We prove that under certain conditions this inevitably leads to a local-optimum barrier, and propose a soft risk mechanism to bypass it. We demonstrate improved risk aversion in maze navigation, autonomous driving, and resource allocation benchmarks.
  arXiv  Detail & Related papers  (2022-05-10T19:40:52Z)
• TOPS: Transition-based VOlatility-controlled Policy Search and its Global Convergence [9.607937067646617]
  This paper proposes Transition-based VOlatility-controlled Policy Search (TOPS) It is a novel algorithm that solves risk-averse problems by learning from (possibly non-consecutive) transitions instead of only consecutive trajectories. Both theoretical analysis and experimental results demonstrate a state-of-the-art level of risk-averse policy search methods.
  arXiv  Detail & Related papers  (2022-01-24T18:29:23Z)
• Learning Sampling Policy for Faster Derivative Free Optimization [100.27518340593284]
  We propose a new reinforcement learning based ZO algorithm (ZO-RL) with learning the sampling policy for generating the perturbations in ZO optimization instead of using random sampling. Our results show that our ZO-RL algorithm can effectively reduce the variances of ZO gradient by learning a sampling policy, and converge faster than existing ZO algorithms in different scenarios.
  arXiv  Detail & Related papers  (2021-04-09T14:50:59Z)
• A Regret Minimization Approach to Iterative Learning Control [61.37088759497583]
  We propose a new performance metric, planning regret, which replaces the standard uncertainty assumptions with worst case regret. We provide theoretical and empirical evidence that the proposed algorithm outperforms existing methods on several benchmarks.
  arXiv  Detail & Related papers  (2021-02-26T13:48:49Z)
• Risk-Sensitive Reinforcement Learning: Near-Optimal Risk-Sample Tradeoff in Regret [115.85354306623368]
  We study risk-sensitive reinforcement learning in episodic Markov decision processes with unknown transition kernels. We propose two provably efficient model-free algorithms, Risk-Sensitive Value Iteration (RSVI) and Risk-Sensitive Q-learning (RSQ) We prove that RSVI attains an $tildeObig(lambda(|beta| H2) cdot sqrtH3 S2AT big)$ regret, while RSQ attains an $tildeObig(lambda
  arXiv  Detail & Related papers  (2020-06-22T19:28:26Z)
• Reparameterized Variational Divergence Minimization for Stable Imitation [57.06909373038396]
  We study the extent to which variations in the choice of probabilistic divergence may yield more performant ILO algorithms. We contribute a re parameterization trick for adversarial imitation learning to alleviate the challenges of the promising $f$-divergence minimization framework. Empirically, we demonstrate that our design choices allow for ILO algorithms that outperform baseline approaches and more closely match expert performance in low-dimensional continuous-control tasks.
  arXiv  Detail & Related papers  (2020-06-18T19:04:09Z)
• Learning Bounds for Risk-sensitive Learning [86.50262971918276]
  In risk-sensitive learning, one aims to find a hypothesis that minimizes a risk-averse (or risk-seeking) measure of loss. We study the generalization properties of risk-sensitive learning schemes whose optimand is described via optimized certainty equivalents.
  arXiv  Detail & Related papers  (2020-06-15T05:25:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.