Global Convergence of Wasserstein Policy Gradient for Entropy-Regularized Reinforcement Learning

• URL: http://arxiv.org/abs/2605.26078v2
• Date: Tue, 26 May 2026 11:46:28 GMT
• Title: Global Convergence of Wasserstein Policy Gradient for Entropy-Regularized Reinforcement Learning
• Abstract summary: Wasserstein policy gradient (WPG) is a policy optimization method for reinforcement learning.<n>We develop a global convergence theory for WPG by exploiting the Bellman structure of entropy-regularized RL.<n>Our analysis shows that although entropy-regularized RL is not convex in the usual flat sense, the Bellman recursion induces a favorable Polyak--Lojasiewicz-type geometry.
• Score: 7.683866416886673
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Wasserstein policy gradient (WPG) is a policy optimization method for reinforcement learning (RL) that exploits the optimal-transport geometry of action distributions. For the entropy-regularized RL objective, WPG evolves each state-conditional policy by transporting it along the action gradient of the soft Q-function together with a Langevin-type diffusion. Despite its appeal for continuous-control problems, its global convergence properties remain poorly understood. Standard Langevin analyses do not directly apply, because the RL objective depends on the policy through the Bellman recursion rather than through a static convex functional, and the Langevin drift is determined by the soft Q-function, whose regularity must be controlled along the policy iterates. In this paper, we develop a global convergence theory for WPG by exploiting the Bellman structure of entropy-regularized RL. We show that the role usually played by convexity can be replaced by a Bellman-based argument: the soft Bellman residual admits a statewise KL representation with respect to a Gibbs policy; Bellman contraction relates this residual to the global optimality gap; and a Bellman resolvent identity connects value improvement to relative Fisher information. Combined with a uniform log-Sobolev inequality (LSI) for the evolving Gibbs family, these ingredients yield a distributional Polyak--Łojasiewicz condition. We further establish the regularity and uniform bounds needed to control the discretization error, thereby obtaining geometric contraction up to a discretization bias. Conceptually, our analysis shows that although entropy-regularized RL is not convex in the usual flat sense, the Bellman recursion induces a favorable Polyak--Lojasiewicz-type (PL) geometry that supports global convergence of WPG.

Related papers

• Wasserstein Policy Gradient for Entropy-Regularized Linear-Quadratic Control [7.683866416886673]
  Wasserstein policy gradient (WPG) updates state-conditional action laws by transport in the action space.<n>We study entropy-regularized discounted linear-quadratic (LQ) control.
  arXiv  Detail & Related papers  (2026-08-07T17:17:58Z)
• Deep Q-Learning on Hölder Spaces [0.5671339754416556]
  We study the operator-theoretic core of Q-learning in continuous-time control with continuous states and actions.<n>We show that a Bellman update maps bounded inputs into an anisotropic regularity class, smoothing the state variable while leaving only Lipschitz dependence on the action variable.<n>The resulting theory makes a direct contribution to Q-learning theory at the level of Bellman target regularity and approximation in continuous control.
  arXiv  Detail & Related papers  (2026-06-15T15:23:22Z)
• Stochastic MeanFlow Policies: One-Step Generative Control with Entropic Mirror Descent [53.828537014796574]
  Online off-policy reinforcement learning (RL) is shaped by two choices: the policy class and the update rule.<n>We propose MeanFlow Policies, a one-step generative policy class that maps noise to actions through a MeanFlow transformation.<n>Across seven MuJoCo benchmarks, S improves over Gaussian and generative baselines while retaining single-step inference efficiency.
  arXiv  Detail & Related papers  (2026-05-20T15:14:14Z)
• Continuous-time q-learning for mean-field control with common noise, part-I: Theoretical foundations [7.003100820506273]
  We investigate the continuous-time counterpart of the Q-function for entropy-regularized mean-field control (MFC) with controlled common noise.<n>We derive the exploratory Hamilton-Jacobi-Bellman (HJB) equation, in which the controlled common noise gives rise to an additional nonlinear functional of policy.<n>In the mean-field setting, we introduce the integrated q-function (Iq-function) defined on the state distribution and the policy, and it is shown that an optimal policy is identified as a two-layer fixed point to the argmax operator of the Iq-function
  arXiv  Detail & Related papers  (2026-04-30T03:37:55Z)
• Stability and Generalization of Push-Sum Based Decentralized Optimization over Directed Graphs [55.77845440440496]
  Push-based decentralized communication enables optimization over communication networks, where information exchange may be asymmetric.<n>We develop a unified uniform-stability framework for the Gradient Push (SGP) algorithm.<n>A key technical ingredient is an imbalance-aware generalization bound through two quantities.
  arXiv  Detail & Related papers  (2026-02-24T05:32:03Z)
• Stationary Reweighting Yields Local Convergence of Soft Fitted Q-Iteration [40.322273308230606]
  We show that fitted Q-iteration and its entropy-regularized variant, soft FQI, behave poorly under function approximation and distribution shift.<n>We introduce stationary-reweighted soft FQI, which reweights each regression update using the stationary distribution of the current policy.<n>Our analysis suggests that global convergence may be recovered by gradually reducing the softmax temperature.
  arXiv  Detail & Related papers  (2025-12-30T00:58:35Z)
• Non-Asymptotic Global Convergence of PPO-Clip [23.221917827987625]
  This paper advances the theoretical foundations of the PPO-Clip algorithm by analyzing a deterministic actor-only PPO algorithm within the general RL setting.<n>We derive a non-uniform Lipschitz smoothness condition and a ojasiewicz inequality for the considered problem.
  arXiv  Detail & Related papers  (2025-12-18T14:06:37Z)
• The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models [99.98293908799731]
  This paper aims to overcome a major obstacle in scaling RL for reasoning with LLMs, namely the collapse of policy entropy.<n>In practice, we establish a transformation equation R=-a*eH+b between entropy H and downstream performance R.<n>We propose two simple yet effective techniques, namely Clip-Cov and KL-Cov, which clip and apply KL penalty to tokens with high covariances respectively.
  arXiv  Detail & Related papers  (2025-05-28T17:38:45Z)
• Rethinking the Global Convergence of Softmax Policy Gradient with Linear Function Approximation [52.772454746132276]
  We show that the approximation error in modeling problem-dependent quantities is irrelevant to the algorithm's global convergence.<n>We prove that $textttLin-SPG$ with any arbitrary constant learning rate can ensure global convergence to the optimal policy.
  arXiv  Detail & Related papers  (2025-05-06T04:03:06Z)
• Entropy annealing for policy mirror descent in continuous time and space [2.8255028200738455]
  We study the impact of entropy regularization on the convergence of policy methods for exit time control problems.<n>This paper explains how entropy regularization improves policy optimization, even with the true gradient.
  arXiv  Detail & Related papers  (2024-05-30T17:02:18Z)
• Regularization Guarantees Generalization in Bayesian Reinforcement Learning through Algorithmic Stability [48.62272919754204]
  We study generalization in Bayesian RL under the probably approximately correct (PAC) framework. Our main contribution is showing that by adding regularization, the optimal policy becomes stable in an appropriate sense.
  arXiv  Detail & Related papers  (2021-09-24T07:48:34Z)
• Policy Mirror Descent for Regularized Reinforcement Learning: A Generalized Framework with Linear Convergence [60.20076757208645]
  This paper proposes a general policy mirror descent (GPMD) algorithm for solving regularized RL. We demonstrate that our algorithm converges linearly over an entire range learning rates, in a dimension-free fashion, to the global solution.
  arXiv  Detail & Related papers  (2021-05-24T02:21:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.