A note on convergence of Wasserstein policy optimization

• URL: http://arxiv.org/abs/2605.22622v1
• Date: Thu, 21 May 2026 15:32:39 GMT
• Title: A note on convergence of Wasserstein policy optimization
• Abstract summary: Wasserstein Policy Optimization (WPO) is a proposed reinforcement learning algorithm that leverages Wasserstein gradient flows to optimize policies in continuous action spaces.<n>We argue that WPO within the framework of entropy-regularised Markov Decision Processes converges linearly.
• Score: 1.4576165959001435
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Wasserstein Policy Optimization (WPO) is a recently proposed reinforcement learning algorithm that leverages Wasserstein gradient flows to optimize stochastic policies in continuous action spaces. Despite its empirical success, the theoretical convergence properties of WPO in environments with continuous state and action spaces have yet to be fully established. In this note, we argue that WPO within the framework of entropy-regularised Markov Decision Processes converges linearly. This is done by leveraging recent advances in mean-field analysis for convergence of gradient flows using log-Sobole inequalities. Assuming existence of sufficiently regular solution to the gradient flow equation we demonstrate monotonic energy dissipation along the flow and establish a local log-Sobolev inequality. Ultimately, these properties allow us to argue that the value function should converge linearly to the global optimum.

Related papers

• Q-Flow: Stable and Expressive Reinforcement Learning with Flow-Based Policy [51.31164293751206]
  We introduce Q-Flow, a framework that leverages the deterministic nature of flow dynamics to explicitly propagate latent states along policy-induced flow.<n>We evaluate Q-Flow in the offline learning setting on the challenging OGBench suite, where it consistently outperforms state-of-the-art baselines by an average of 10.6 percentage points.
  arXiv  Detail & Related papers  (2026-05-13T12:31:02Z)
• Achieve Performatively Optimal Policy for Performative Reinforcement Learning [55.983627302691424]
  This work proposes a zeroth-order FrankWolfe- (0FW) algorithm with a gradient of performative policy in the framework.<n> Experimental results demonstrate that our 0FW is more effective than the existing approximation in finding the desired PO policy.
  arXiv  Detail & Related papers  (2025-10-06T01:56:31Z)
• Remarks on the Polyak-Lojasiewicz inequality and the convergence of gradient systems [0.3277163122167434]
  This work explores generalizations of the Polyak-Lojasiewicz inequality (PLI)<n>This work shows that while weaker conditions are sufficient for global convergence to, and optimality of the set of critical points of the cost function, the "profile" of the gradient flow solution can change significantly depending on which "flavor" of inequality the cost satisfies.
  arXiv  Detail & Related papers  (2025-03-31T00:59:56Z)
• Wasserstein Distributionally Robust Bayesian Optimization with Continuous Context [5.4147994801415145]
  We address the challenge of sequential data-driven decision-making under context distributional uncertainty.<n>We propose a novel algorithm for Wasserstein Distributionally Robust Bayesian Optimization.<n>Our theoretical analysis combines recent results in self-normalized concentration in Hilbert spaces and finite-sample bounds for distributionally robust optimization.
  arXiv  Detail & Related papers  (2025-03-26T09:11:17Z)
• Entropy annealing for policy mirror descent in continuous time and space [2.8255028200738455]
  We study the impact of entropy regularization on the convergence of policy methods for exit time control problems.<n>This paper explains how entropy regularization improves policy optimization, even with the true gradient.
  arXiv  Detail & Related papers  (2024-05-30T17:02:18Z)
• On the Global Convergence of Policy Gradient in Average Reward Markov Decision Processes [50.68789924454235]
  We present the first finite time global convergence analysis of policy gradient in the context of average reward Markov decision processes (MDPs) Our analysis shows that the policy gradient iterates converge to the optimal policy at a sublinear rate of $Oleft(frac1Tright),$ which translates to $Oleft(log(T)right)$ regret, where $T$ represents the number of iterations.
  arXiv  Detail & Related papers  (2024-03-11T15:25:03Z)
• A Fisher-Rao gradient flow for entropy-regularised Markov decision processes in Polish spaces [10.045995853506222]
  We study the global convergence of a Fisher-Rao policy gradient flow for infinite-horizon entropy-regularised Markov decision processes with Polish state and action space.<n>We establish the global well-posedness of the gradient flow and demonstrate its exponential convergence to the optimal policy.
  arXiv  Detail & Related papers  (2023-10-04T16:41:36Z)
• Convergence of Adam Under Relaxed Assumptions [72.24779199744954]
  We show that Adam converges to $epsilon$-stationary points with $O(epsilon-4)$ gradient complexity under far more realistic conditions. We also propose a variance-reduced version of Adam with an accelerated gradient complexity of $O(epsilon-3)$.
  arXiv  Detail & Related papers  (2023-04-27T06:27:37Z)
• The Role of Baselines in Policy Gradient Optimization [83.42050606055822]
  We show that the emphstate value baseline allows on-policy. emphnatural policy gradient (NPG) to converge to a globally optimal. policy at an $O (1/t) rate gradient. We find that the primary effect of the value baseline is to textbfreduce the aggressiveness of the updates rather than their variance.
  arXiv  Detail & Related papers  (2023-01-16T06:28:00Z)
• Global Convergence of Over-parameterized Deep Equilibrium Models [52.65330015267245]
  A deep equilibrium model (DEQ) is implicitly defined through an equilibrium point of an infinite-depth weight-tied model with an input-injection. Instead of infinite computations, it solves an equilibrium point directly with root-finding and computes gradients with implicit differentiation. We propose a novel probabilistic framework to overcome the technical difficulty in the non-asymptotic analysis of infinite-depth weight-tied models.
  arXiv  Detail & Related papers  (2022-05-27T08:00:13Z)
• Convergence of policy gradient for entropy regularized MDPs with neural network approximation in the mean-field regime [0.0]
  We study the global convergence of policy gradient for infinite-horizon, continuous state and action space, entropy-regularized Markov decision processes (MDPs) Our results rely on the careful analysis of non-linear Fokker--Planck--Kolmogorov equation.
  arXiv  Detail & Related papers  (2022-01-18T20:17:16Z)
• On the Convergence of Stochastic Extragradient for Bilinear Games with Restarted Iteration Averaging [96.13485146617322]
  We present an analysis of the ExtraGradient (SEG) method with constant step size, and present variations of the method that yield favorable convergence. We prove that when augmented with averaging, SEG provably converges to the Nash equilibrium, and such a rate is provably accelerated by incorporating a scheduled restarting procedure.
  arXiv  Detail & Related papers  (2021-06-30T17:51:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.