Second-Order Actor-Critic Methods for Discounted MDPs via Policy Hessian Decomposition

• URL: http://arxiv.org/abs/2605.14982v1
• Date: Thu, 14 May 2026 15:46:27 GMT
• Title: Second-Order Actor-Critic Methods for Discounted MDPs via Policy Hessian Decomposition
• Abstract summary: We analyze second-order approximations for the actor update that leverage the full curvature information of the objective.<n>We show that this approximation becomes well-justified under a two-timescale actor-critic framework.<n>Building on this insight, we formulate a second-order actor-critic method for the discounted reward setting.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We address the discounted reward setting in reinforcement learning (RL). To mitigate the value approximation challenges in policy gradient methods, actor-critic approaches have been developed and are known to converge to stationary points under suitable assumptions. However, these methods rely on first-order updates. In contrast, second-order optimization provides principled curvature-aware updates that are proven to accelerate convergence, but its application in RL is limited by the computational complexity of Hessian estimation. In this work, we analyze second-order approximations for the actor update that leverage the full curvature information of the objective as much as possible. A stable approximation requires treating the action-value function as locally constant with respect to policy parameters, which does not generally hold in policy gradient methods. We show that this approximation becomes well-justified under a two-timescale actor-critic framework, where the critic evolves on a faster timescale and can be treated as quasi-stationary during actor updates. Building on this insight, we formulate a second-order actor-critic method for the discounted reward setting that leverages Hessian-vector product (HVP) computations, resulting in a computationally efficient and stable second-order update.

Related papers

• Actor-Critic with Active Importance Sampling [61.41272490044518]
  This paper introduces the Active-Importance-Sampling Actor-Critic (AISAC) algorithm.<n>AISAC is an extension of the Actor-Critic framework for reducing variance in policy gradient estimation.<n> Experiments show improved learning speed, sample efficiency, and training compared to standard Actor-Critic methods.
  arXiv  Detail & Related papers  (2026-05-08T01:21:32Z)
• Quasi-Newton Compatible Actor-Critic for Deterministic Policies [2.745166507489733]
  We propose a second-order deterministic actor-critic framework in reinforcement learning.<n>We introduce a quadratic critic that simultaneously preserves the true policy gradient and an approximation of the performance Hessian.
  arXiv  Detail & Related papers  (2025-11-12T17:49:02Z)
• Policy Gradient with Second Order Momentum [2.44755919161855]
  Policy Gradient with Second-Order Momentum (PG-SOM) is a lightweight second-order optimisation scheme for reinforcement-learning policies.<n>PG-SOM augments the classical REINFORCE update with two exponentially weighted statistics: a first-order gradient average and a diagonal approximation of the Hessian.<n>Experiments on standard control benchmarks show up to a 2.1x increase in sample efficiency and a substantial reduction in variance compared to first-order and Fisher-matrix baselines.
  arXiv  Detail & Related papers  (2025-05-16T06:23:53Z)
• On the Second-Order Convergence of Biased Policy Gradient Algorithms [11.955062839855334]
  gradient policy escapes saddle at second-order stationary points. We provide a novel second-order analysis of biased gradient methods. We also establish the convergence points on chains initial state distribution.
  arXiv  Detail & Related papers  (2023-11-05T02:33:30Z)
• A Two-Time-Scale Stochastic Optimization Framework with Applications in Control and Reinforcement Learning [13.908826484332282]
  We study a new two-time-scale gradient method for solving optimization problems. Our first contribution is to characterize the finite-time complexity of the proposed two-time-scale gradient algorithm. We apply our framework to gradient-based policy evaluation algorithms in reinforcement learning.
  arXiv  Detail & Related papers  (2021-09-29T23:15:23Z)
• Doubly Robust Off-Policy Actor-Critic: Convergence and Optimality [131.45028999325797]
  We develop a doubly robust off-policy AC (DR-Off-PAC) for discounted MDP. DR-Off-PAC adopts a single timescale structure, in which both actor and critics are updated simultaneously with constant stepsize. We study the finite-time convergence rate and characterize the sample complexity for DR-Off-PAC to attain an $epsilon$-accurate optimal policy.
  arXiv  Detail & Related papers  (2021-02-23T18:56:13Z)
• Variance Penalized On-Policy and Off-Policy Actor-Critic [60.06593931848165]
  We propose on-policy and off-policy actor-critic algorithms that optimize a performance criterion involving both mean and variance in the return. Our approach not only performs on par with actor-critic and prior variance-penalization baselines in terms of expected return, but also generates trajectories which have lower variance in the return.
  arXiv  Detail & Related papers  (2021-02-03T10:06:16Z)
• Average-Reward Off-Policy Policy Evaluation with Function Approximation [66.67075551933438]
  We consider off-policy policy evaluation with function approximation in average-reward MDPs. bootstrapping is necessary and, along with off-policy learning and FA, results in the deadly triad. We propose two novel algorithms, reproducing the celebrated success of Gradient TD algorithms in the average-reward setting.
  arXiv  Detail & Related papers  (2021-01-08T00:43:04Z)
• Single-Timescale Actor-Critic Provably Finds Globally Optimal Policy [122.01837436087516]
  We study the global convergence and global optimality of actor-critic, one of the most popular families of reinforcement learning algorithms. We establish the rate of convergence and global optimality of single-timescale actor-critic with linear function approximation for the first time.
  arXiv  Detail & Related papers  (2020-08-02T14:01:49Z)
• Kalman meets Bellman: Improving Policy Evaluation through Value Tracking [59.691919635037216]
  Policy evaluation is a key process in Reinforcement Learning (RL) We devise an optimization method, called Kalman Optimization for Value Approximation (KOVA) KOVA minimizes a regularized objective function that concerns both parameter and noisy return uncertainties.
  arXiv  Detail & Related papers  (2020-02-17T13:30:43Z)
• Greedy Actor-Critic: A New Conditional Cross-Entropy Method for Policy Improvement [31.602912612167856]
  In this work, we explore an alternative update for the actor, based on an extension of the cross entropy method (CEM) to condition on inputs (states) The speed of this concentration is controlled by a proposal policy, that concentrates at a slower rate than the actor. We empirically show that our Greedy AC algorithm, that uses CCEM for the actor update, performs better than Soft Actor-Critic and is much less sensitive to entropy-regularization.
  arXiv  Detail & Related papers  (2018-10-22T06:35:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.