GPG: Generalized Policy Gradient Theorem for Transformer-based Policies

• URL: http://arxiv.org/abs/2512.10365v1
• Date: Thu, 11 Dec 2025 07:30:33 GMT
• Title: GPG: Generalized Policy Gradient Theorem for Transformer-based Policies
• Abstract summary: We present the Generalized Policy Gradient (GPG) Theorem, specifically designed for Transformer-based policies.<n> Notably, we demonstrate that both standard Policy Gradient Theorem and GRPO emerge as special cases within our GPG framework.
• Score: 53.200540736223395
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present the Generalized Policy Gradient (GPG) Theorem, specifically designed for Transformer-based policies. Notably, we demonstrate that both standard Policy Gradient Theorem and GRPO emerge as special cases within our GPG framework. Furthermore, we explore its practical applications in training Large Language Models (LLMs), offering new insights into efficient policy optimization.

Related papers

• OGPO: Sample Efficient Full-Finetuning of Generative Control Policies [53.42266064673132]
  Generative control policies (GCPs) have emerged as effective parameterizations for robot learning.<n>This work introduces Off-policy Generative Policy Optimization (OGPO), a sample-efficient for finetuning GCPs.<n>OGPO achieves state-of-the-art performance on manipulation tasks spanning multi-task settings, high-precision insertion, and dexterous control.
  arXiv  Detail & Related papers  (2026-05-04T18:36:40Z)
• Demystifying Group Relative Policy Optimization: Its Policy Gradient is a U-Statistic [12.256817975993128]
  Relative policy optimization is a core methodological component of DeepSeekMath and DeepSeek-R1.<n>This paper provides a unified framework to understand GRPO through the lens of classical U-statistics.
  arXiv  Detail & Related papers  (2026-03-01T15:56:43Z)
• Residual Policy Gradient: A Reward View of KL-regularized Objective [48.39829592175419]
  Reinforcement Learning and Imitation Learning have achieved widespread success in many domains but remain constrained during real-world deployment.<n>Policy customization has been introduced, aiming to adapt a prior policy while preserving its inherent properties and meeting new task-specific requirements.<n>A principled approach to policy customization is Residual Q-Learning (RQL), which formulates the problem as a Markov Decision Process (MDP) and derives a family of value-based learning algorithms.<n>We introduce Residual Policy Gradient (RPG), which extends RQL to policy gradient methods, allowing policy customization in gradient-based RL settings.
  arXiv  Detail & Related papers  (2025-03-14T02:30:13Z)
• Convergence of Policy Mirror Descent Beyond Compatible Function Approximation [66.4260157478436]
  We develop theoretical PMD general policy classes where we strictly assume a weaker variational dominance and obtain convergence to the best-in-class policy.<n>Our main notion leverages a novel notion induced by the local norm induced by the occupancy- gradient measure.
  arXiv  Detail & Related papers  (2025-02-16T08:05:46Z)
• On the Global Optimality of Policy Gradient Methods in General Utility Reinforcement Learning [30.767979998925437]
  Reinforcement learning with general utilities (RLGU) offers a unifying framework to capture problems beyond standard expected returns.<n>Recent advances in theoretical analysis of policy gradient (PG) methods for standard RL and recent efforts in RLGU still remain limited.<n>We establish global optimality guarantees of PG methods for RLGU in which the objective is a general concave utility function of the state-action occupancy measure.
  arXiv  Detail & Related papers  (2024-10-05T10:24:07Z)
• Last-Iterate Global Convergence of Policy Gradients for Constrained Reinforcement Learning [62.81324245896717]
  We introduce an exploration-agnostic algorithm, called C-PG, which exhibits global last-ite convergence guarantees under (weak) gradient domination assumptions. We numerically validate our algorithms on constrained control problems, and compare them with state-of-the-art baselines.
  arXiv  Detail & Related papers  (2024-07-15T14:54:57Z)
• Matryoshka Policy Gradient for Entropy-Regularized RL: Convergence and Global Optimality [0.5261718469769449]
  A novel Policy Gradient (PG) algorithm, called $textitMatryoshka Policy Gradient$ (MPG), is introduced and studied. We prove uniqueness and characterize the optimal policy of the entropy regularized objective, together with global convergence of MPG. As a proof of concept, we evaluate numerically MPG on standard test benchmarks.
  arXiv  Detail & Related papers  (2023-03-22T17:56:18Z)
• A Novel Framework for Policy Mirror Descent with General Parameterization and Linear Convergence [15.807079236265714]
  We introduce a novel framework for policy optimization based on mirror descent. We obtain the first result that guarantees linear convergence for a policy-gradient-based method involving general parameterization.
  arXiv  Detail & Related papers  (2023-01-30T18:21:48Z)
• Generalised Policy Improvement with Geometric Policy Composition [18.80807234471197]
  We introduce a method for policy improvement that interpolates between the greedy approach of value-based reinforcement learning (RL) and the full planning approach typical of model-based RL. We show that we can evaluate any non-Markov policy that switches between a set of base Markov policies with fixed probability by a careful composition of the base policy GHMs. We can then apply generalised policy improvement (GPI) to collections of such non-Markov policies to obtain a new Markov policy that will in general outperform its precursors.
  arXiv  Detail & Related papers  (2022-06-17T12:52:13Z)
• When Will Generative Adversarial Imitation Learning Algorithms Attain Global Convergence [56.40794592158596]
  We study generative adversarial imitation learning (GAIL) under general MDP and for nonlinear reward function classes. This is the first systematic theoretical study of GAIL for global convergence.
  arXiv  Detail & Related papers  (2020-06-24T06:24:37Z)
• Representation of Reinforcement Learning Policies in Reproducing Kernel Hilbert Spaces [72.5149277196468]
  This framework involves finding a low-dimensional embedding of the policy on a kernel Hilbert space (RKHS) We derive strong theoretical guarantees on the expected return of the reconstructed policy. The results confirm that the policies can be robustly embedded in a low-dimensional space while the embedded policy incurs almost no decrease in return.
  arXiv  Detail & Related papers  (2020-02-07T15:57:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.