Extending Group Relative Policy Optimization to Continuous Control: A Theoretical Framework for Robotic Reinforcement Learning

• URL: http://arxiv.org/abs/2507.19555v1
• Date: Fri, 25 Jul 2025 05:25:40 GMT
• Title: Extending Group Relative Policy Optimization to Continuous Control: A Theoretical Framework for Robotic Reinforcement Learning
• Abstract summary: Group Relative Policy Optimization (GRPO) has shown promise in discrete action spaces by eliminating value function dependencies through group-based advantage estimation.<n>This paper presents a theoretical framework extending GRPO to continuous control environments, addressing challenges in high-dimensional action spaces, sparse rewards, and temporal dynamics.
• Score: 0.0
• License: http://creativecommons.org/publicdomain/zero/1.0/
• Abstract: Group Relative Policy Optimization (GRPO) has shown promise in discrete action spaces by eliminating value function dependencies through group-based advantage estimation. However, its application to continuous control remains unexplored, limiting its utility in robotics where continuous actions are essential. This paper presents a theoretical framework extending GRPO to continuous control environments, addressing challenges in high-dimensional action spaces, sparse rewards, and temporal dynamics. Our approach introduces trajectory-based policy clustering, state-aware advantage estimation, and regularized policy updates designed for robotic applications. We provide theoretical analysis of convergence properties and computational complexity, establishing a foundation for future empirical validation in robotic systems including locomotion and manipulation tasks.

Related papers

• Guided Streaming Stochastic Interpolant Policy [8.673777984856075]
  Inference-time guidance is essential for steering generative robot policies toward dynamic objectives without retraining.<n>In this work, we derive the optimal guidance term for dynamic, unstructured Interpolants (SI)<n>We apply this framework to real-time control through the Streaming Interpolant Policy (SSIP)
  arXiv  Detail & Related papers  (2026-05-11T06:24:21Z)
• OGPO: Sample Efficient Full-Finetuning of Generative Control Policies [53.42266064673132]
  Generative control policies (GCPs) have emerged as effective parameterizations for robot learning.<n>This work introduces Off-policy Generative Policy Optimization (OGPO), a sample-efficient for finetuning GCPs.<n>OGPO achieves state-of-the-art performance on manipulation tasks spanning multi-task settings, high-precision insertion, and dexterous control.
  arXiv  Detail & Related papers  (2026-05-04T18:36:40Z)
• Constrained Group Relative Policy Optimization [18.3888203751956]
  We introduce Constrained GRPO, a Lagrangian-based extension of GRPO for constrained policy optimization.<n>We show that a naive multi-component treatment in advantage estimation can break constrained learning.<n>We also evaluate Constrained GRPO on robotics tasks, where it improves constraint satisfaction while increasing task success.
  arXiv  Detail & Related papers  (2026-02-05T16:44:23Z)
• GPO: Growing Policy Optimization for Legged Robot Locomotion and Whole-Body Control [20.055658070161183]
  Training reinforcement learning (RL) policies for legged robots remains challenging due to high-dimensional continuous actions, hardware constraints, and limited exploration.<n>We introduce Growing Policy Optimization (GPO), a training framework that applies a time-varying action transformation to restrict the effective action space in the early stage.<n>We evaluate GPO on both quadruped and hexapod robots, including zero-shot deployment of simulation-trained policies on hardware.
  arXiv  Detail & Related papers  (2026-01-28T14:49:52Z)
• MAESTRO: Meta-learning Adaptive Estimation of Scalarization Trade-offs for Reward Optimization [56.074760766965085]
  Group-Relative Policy Optimization has emerged as an efficient paradigm for aligning Large Language Models (LLMs)<n>We propose MAESTRO, which treats reward scalarization as a dynamic latent policy, leveraging the model's terminal hidden states as a semantic bottleneck.<n>We formulate this as a contextual bandit problem within a bi-level optimization framework, where a lightweight Conductor network co-evolves with the policy by utilizing group-relative advantages as a meta-reward signal.
  arXiv  Detail & Related papers  (2026-01-12T05:02:48Z)
• Learning Deterministic Policies with Policy Gradients in Constrained Markov Decision Processes [59.27926064817273]
  We introduce an exploration-agnostic algorithm, called C-PG, which enjoys global last-iterate convergence guarantees under domination assumptions.<n>We empirically validate both the action-based (C-PGAE) and parameter-based (C-PGPE) variants of C-PG on constrained control tasks.
  arXiv  Detail & Related papers  (2025-06-06T10:29:05Z)
• COMBO-Grasp: Learning Constraint-Based Manipulation for Bimanual Occluded Grasping [56.907940167333656]
  Occluded robot grasping is where the desired grasp poses are kinematically infeasible due to environmental constraints such as surface collisions.<n>Traditional robot manipulation approaches struggle with the complexity of non-prehensile or bimanual strategies commonly used by humans.<n>We introduce Constraint-based Manipulation for Bimanual Occluded Grasping (COMBO-Grasp), a learning-based approach which leverages two coordinated policies.
  arXiv  Detail & Related papers  (2025-02-12T01:31:01Z)
• Contractive Dynamical Imitation Policies for Efficient Out-of-Sample Recovery [3.549243565065057]
  Imitation learning is a data-driven approach to learning policies from expert behavior.<n>It is prone to unreliable outcomes in out-of-sample (OOS) regions.<n>We propose a framework for learning policies modeled by contractive dynamical systems.
  arXiv  Detail & Related papers  (2024-12-10T14:28:18Z)
• Learning Maximal Safe Sets Using Hypernetworks for MPC-based Local Trajectory Planning in Unknown Environments [1.3182466374784207]
  This paper presents a novel learning-based approach for online estimation of maximal safe sets for local trajectory planning in unknown static environments.<n>The neural representation of a set is used as the terminal set constraint for a model predictive control (MPC) local planner.<n>We deploy our proposed method, NTC-MPC, on a physical robot and demonstrate its ability to safely avoid obstacles in scenarios where the baselines fail.
  arXiv  Detail & Related papers  (2024-10-26T20:37:57Z)
• Last-Iterate Global Convergence of Policy Gradients for Constrained Reinforcement Learning [62.81324245896717]
  We introduce an exploration-agnostic algorithm, called C-PG, which exhibits global last-ite convergence guarantees under (weak) gradient domination assumptions. We numerically validate our algorithms on constrained control problems, and compare them with state-of-the-art baselines.
  arXiv  Detail & Related papers  (2024-07-15T14:54:57Z)
• Integrating DeepRL with Robust Low-Level Control in Robotic Manipulators for Non-Repetitive Reaching Tasks [0.24578723416255746]
  In robotics, contemporary strategies are learning-based, characterized by a complex black-box nature and a lack of interpretability. We propose integrating a collision-free trajectory planner based on deep reinforcement learning (DRL) with a novel auto-tuning low-level control strategy.
  arXiv  Detail & Related papers  (2024-02-04T15:54:03Z)
• Multi-Agent Dynamic Relational Reasoning for Social Robot Navigation [50.01551945190676]
  Social robot navigation can be helpful in various contexts of daily life but requires safe human-robot interactions and efficient trajectory planning. We propose a systematic relational reasoning approach with explicit inference of the underlying dynamically evolving relational structures. We demonstrate its effectiveness for multi-agent trajectory prediction and social robot navigation.
  arXiv  Detail & Related papers  (2024-01-22T18:58:22Z)
• Learning Robust Policies for Generalized Debris Capture with an Automated Tether-Net System [2.0429716172112617]
  This paper presents a reinforcement learning framework that integrates a policy optimization approach with net dynamics simulations. A state transition model is considered in order to incorporate synthetic uncertainties in state estimation and launch actuation. The trained policy demonstrates capture performance close to that obtained with reliability-based optimization run over an individual scenario.
  arXiv  Detail & Related papers  (2022-01-11T20:09:05Z)
• Guided Constrained Policy Optimization for Dynamic Quadrupedal Robot Locomotion [78.46388769788405]
  We introduce guided constrained policy optimization (GCPO), an RL framework based upon our implementation of constrained policy optimization (CPPO) We show that guided constrained RL offers faster convergence close to the desired optimum resulting in an optimal, yet physically feasible, robotic control behavior without the need for precise reward function tuning.
  arXiv  Detail & Related papers  (2020-02-22T10:15:53Z)
• Discrete Action On-Policy Learning with Action-Value Critic [72.20609919995086]
  Reinforcement learning (RL) in discrete action space is ubiquitous in real-world applications, but its complexity grows exponentially with the action-space dimension. We construct a critic to estimate action-value functions, apply it on correlated actions, and combine these critic estimated action values to control the variance of gradient estimation. These efforts result in a new discrete action on-policy RL algorithm that empirically outperforms related on-policy algorithms relying on variance control techniques.
  arXiv  Detail & Related papers  (2020-02-10T04:23:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.