Risk-Sensitive Q-Learning in Continuous Time with Application to Dynamic Portfolio Selection

• URL: http://arxiv.org/abs/2512.02386v1
• Date: Tue, 02 Dec 2025 03:54:53 GMT
• Title: Risk-Sensitive Q-Learning in Continuous Time with Application to Dynamic Portfolio Selection
• Authors: Chuhan Xie,
• Abstract summary: We prove that when the functional is an optimized certainty equivalent (OCE), the optimal policy is Markovian with respect to an augmented environment.<n>We also propose textitCT-RS-q, a risk-sensitive q-learning algorithm based on a novel martingale characterization approach.
• Score: 2.8766374696553823
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper studies the problem of risk-sensitive reinforcement learning (RSRL) in continuous time, where the environment is characterized by a controllable stochastic differential equation (SDE) and the objective is a potentially nonlinear functional of cumulative rewards. We prove that when the functional is an optimized certainty equivalent (OCE), the optimal policy is Markovian with respect to an augmented environment. We also propose \textit{CT-RS-q}, a risk-sensitive q-learning algorithm based on a novel martingale characterization approach. Finally, we run a simulation study on a dynamic portfolio selection problem and illustrate the effectiveness of our algorithm.

Related papers

• Risk-sensitive Reinforcement Learning Based on Convex Scoring Functions [8.758206783988404]
  We propose a reinforcement learning framework under a broad class of risk objectives, characterized by convex scoring functions.<n>This class covers many common risk measures, such as variance, Expected Shortfall, entropic Value-at-Risk, and mean-risk utility.<n>We validate our approach in simulation experiments with a financial application in statistical arbitrage trading, demonstrating the effectiveness of the algorithm.
  arXiv  Detail & Related papers  (2025-05-07T16:31:42Z)
• Efficient Risk-sensitive Planning via Entropic Risk Measures [51.42922439693624]
  We show that only Entropic Risk Measures (EntRM) can be efficiently optimized through dynamic programming.<n>We prove that this optimality front can be computed effectively thanks to a novel structural analysis and smoothness properties of entropic risks.
  arXiv  Detail & Related papers  (2025-02-27T09:56:51Z)
• Robust Reinforcement Learning with Dynamic Distortion Risk Measures [0.0]
  We devise a framework to solve robust risk-aware reinforcement learning problems.<n>We simultaneously account for environmental uncertainty and risk with a class of dynamic robust distortion risk measures.<n>We construct an actor-critic algorithm to solve this class of robust risk-aware RL problems.
  arXiv  Detail & Related papers  (2024-09-16T08:54:59Z)
• Stochastic Q-learning for Large Discrete Action Spaces [79.1700188160944]
  In complex environments with discrete action spaces, effective decision-making is critical in reinforcement learning (RL) We present value-based RL approaches which, as opposed to optimizing over the entire set of $n$ actions, only consider a variable set of actions, possibly as small as $mathcalO(log(n)$)$. The presented value-based RL methods include, among others, Q-learning, StochDQN, StochDDQN, all of which integrate this approach for both value-function updates and action selection.
  arXiv  Detail & Related papers  (2024-05-16T17:58:44Z)
• Fast Value Tracking for Deep Reinforcement Learning [7.648784748888187]
  Reinforcement learning (RL) tackles sequential decision-making problems by creating agents that interact with their environment. Existing algorithms often view these problem as static, focusing on point estimates for model parameters to maximize expected rewards. Our research leverages the Kalman paradigm to introduce a novel quantification and sampling algorithm called Langevinized Kalman TemporalTD.
  arXiv  Detail & Related papers  (2024-03-19T22:18:19Z)
• Model-Based Epistemic Variance of Values for Risk-Aware Policy Optimization [59.758009422067]
  We consider the problem of quantifying uncertainty over expected cumulative rewards in model-based reinforcement learning. We propose a new uncertainty Bellman equation (UBE) whose solution converges to the true posterior variance over values. We introduce a general-purpose policy optimization algorithm, Q-Uncertainty Soft Actor-Critic (QU-SAC) that can be applied for either risk-seeking or risk-averse policy optimization.
  arXiv  Detail & Related papers  (2023-12-07T15:55:58Z)
• Pointer Networks with Q-Learning for Combinatorial Optimization [55.2480439325792]
  We introduce the Pointer Q-Network (PQN), a hybrid neural architecture that integrates model-free Q-value policy approximation with Pointer Networks (Ptr-Nets) Our empirical results demonstrate the efficacy of this approach, also testing the model in unstable environments.
  arXiv  Detail & Related papers  (2023-11-05T12:03:58Z)
• Conditionally Elicitable Dynamic Risk Measures for Deep Reinforcement Learning [0.0]
  We develop an efficient approach to estimate a class of dynamic spectral risk measures with deep neural networks. We also develop a risk-sensitive actor-critic algorithm that uses full episodes and does not require any additional nested transitions.
  arXiv  Detail & Related papers  (2022-06-29T14:11:15Z)
• Reinforcement Learning with Dynamic Convex Risk Measures [0.0]
  We develop an approach for solving time-consistent risk-sensitive optimization problems using model-free reinforcement learning (RL) We employ a time-consistent dynamic programming principle to determine the value of a particular policy, and develop policy gradient update rules.
  arXiv  Detail & Related papers  (2021-12-26T16:41:05Z)
• SAMBA: Safe Model-Based & Active Reinforcement Learning [59.01424351231993]
  SAMBA is a framework for safe reinforcement learning that combines aspects from probabilistic modelling, information theory, and statistics. We evaluate our algorithm on a variety of safe dynamical system benchmarks involving both low and high-dimensional state representations. We provide intuition as to the effectiveness of the framework by a detailed analysis of our active metrics and safety constraints.
  arXiv  Detail & Related papers  (2020-06-12T10:40:46Z)
• Robust Reinforcement Learning with Wasserstein Constraint [49.86490922809473]
  We show the existence of optimal robust policies, provide a sensitivity analysis for the perturbations, and then design a novel robust learning algorithm. The effectiveness of the proposed algorithm is verified in the Cart-Pole environment.
  arXiv  Detail & Related papers  (2020-06-01T13:48:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.