Wasserstein Policy Gradient for Entropy-Regularized Linear-Quadratic Control

• URL: http://arxiv.org/abs/2608.07433v1
• Date: Fri, 07 Aug 2026 17:17:58 GMT
• Title: Wasserstein Policy Gradient for Entropy-Regularized Linear-Quadratic Control
• Abstract summary: Wasserstein policy gradient (WPG) updates state-conditional action laws by transport in the action space.<n>We study entropy-regularized discounted linear-quadratic (LQ) control.
• Score: 7.683866416886673
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Wasserstein policy gradient (WPG) updates state-conditional action laws by transport in the action space. We study entropy-regularized discounted linear-quadratic (LQ) control. A Bellman verification argument shows that the unrestricted problem has a linear-Gaussian optimal policy, and the discounted-occupancy-weighted statewise Wasserstein gradient is tangent to this policy class. WPG therefore reduces exactly to a finite-dimensional ODE for the feedback gain and action covariance. We prove that this ODE is globally well posed and converges exponentially from every admissible initialization. For each fixed LQ problem, the exponent has a positive limit as the entropy temperature tends to zero and contains no perturbative factor of the form $\exp(-c/τ)$, while retaining the usual dependence on the conditioning of the control problem.

Related papers

• Global Convergence of Wasserstein Policy Gradient for Entropy-Regularized Reinforcement Learning [7.683866416886673]
  Wasserstein policy gradient (WPG) is a policy optimization method for reinforcement learning.<n>We develop a global convergence theory for WPG by exploiting the Bellman structure of entropy-regularized RL.<n>Our analysis shows that although entropy-regularized RL is not convex in the usual flat sense, the Bellman recursion induces a favorable Polyak--Lojasiewicz-type geometry.
  arXiv  Detail & Related papers  (2026-05-25T17:42:59Z)
• A note on convergence of Wasserstein policy optimization [1.4576165959001435]
  Wasserstein Policy Optimization (WPO) is a proposed reinforcement learning algorithm that leverages Wasserstein gradient flows to optimize policies in continuous action spaces.<n>We argue that WPO within the framework of entropy-regularised Markov Decision Processes converges linearly.
  arXiv  Detail & Related papers  (2026-05-21T15:32:39Z)
• Achieve Performatively Optimal Policy for Performative Reinforcement Learning [55.983627302691424]
  This work proposes a zeroth-order FrankWolfe- (0FW) algorithm with a gradient of performative policy in the framework.<n> Experimental results demonstrate that our 0FW is more effective than the existing approximation in finding the desired PO policy.
  arXiv  Detail & Related papers  (2025-10-06T01:56:31Z)
• The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models [99.98293908799731]
  This paper aims to overcome a major obstacle in scaling RL for reasoning with LLMs, namely the collapse of policy entropy.<n>In practice, we establish a transformation equation R=-a*eH+b between entropy H and downstream performance R.<n>We propose two simple yet effective techniques, namely Clip-Cov and KL-Cov, which clip and apply KL penalty to tokens with high covariances respectively.
  arXiv  Detail & Related papers  (2025-05-28T17:38:45Z)
• Optimal Rates of Convergence for Entropy Regularization in Discounted Markov Decision Processes [6.415644997939936]
  We study the error introduced by entropy regularization in Markov decision processes.<n>We show that this error decreases exponentially in the inverse regularization strength.<n>We extend our analysis to settings beyond the entropy.
  arXiv  Detail & Related papers  (2024-06-06T15:20:37Z)
• Entropy annealing for policy mirror descent in continuous time and space [2.8255028200738455]
  We study the impact of entropy regularization on the convergence of policy methods for exit time control problems.<n>This paper explains how entropy regularization improves policy optimization, even with the true gradient.
  arXiv  Detail & Related papers  (2024-05-30T17:02:18Z)
• The Role of Baselines in Policy Gradient Optimization [83.42050606055822]
  We show that the emphstate value baseline allows on-policy. emphnatural policy gradient (NPG) to converge to a globally optimal. policy at an $O (1/t) rate gradient. We find that the primary effect of the value baseline is to textbfreduce the aggressiveness of the updates rather than their variance.
  arXiv  Detail & Related papers  (2023-01-16T06:28:00Z)
• Convergence of policy gradient methods for finite-horizon exploratory linear-quadratic control problems [3.8661825615213012]
  We study the global linear convergence of policy gradient (PG) methods for finite-horizon continuous-time exploratory linear-quadratic control (LQC) problems. We propose a novel PG method with discretetime policies. The algorithm leverages the continuous-time analysis, and achieves a robust linear convergence across different action frequencies.
  arXiv  Detail & Related papers  (2022-11-01T17:31:41Z)
• Global Convergence of Over-parameterized Deep Equilibrium Models [52.65330015267245]
  A deep equilibrium model (DEQ) is implicitly defined through an equilibrium point of an infinite-depth weight-tied model with an input-injection. Instead of infinite computations, it solves an equilibrium point directly with root-finding and computes gradients with implicit differentiation. We propose a novel probabilistic framework to overcome the technical difficulty in the non-asymptotic analysis of infinite-depth weight-tied models.
  arXiv  Detail & Related papers  (2022-05-27T08:00:13Z)
• Deep Learning Approximation of Diffeomorphisms via Linear-Control Systems [91.3755431537592]
  We consider a control system of the form $dot x = sum_i=1lF_i(x)u_i$, with linear dependence in the controls. We use the corresponding flow to approximate the action of a diffeomorphism on a compact ensemble of points.
  arXiv  Detail & Related papers  (2021-10-24T08:57:46Z)
• Approximation Schemes for ReLU Regression [80.33702497406632]
  We consider the fundamental problem of ReLU regression. The goal is to output the best fitting ReLU with respect to square loss given to draws from some unknown distribution.
  arXiv  Detail & Related papers  (2020-05-26T16:26:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.