ContractionPPO: Certified Reinforcement Learning via Differentiable Contraction Layers

• URL: http://arxiv.org/abs/2603.19632v1
• Date: Fri, 20 Mar 2026 04:32:18 GMT
• Title: ContractionPPO: Certified Reinforcement Learning via Differentiable Contraction Layers
• Abstract summary: We present ContractionPPO, a framework for robust planning and control of legged robots.<n>ContractionPPO produces a contraction metric that certifies exponential stability of the simulated closed-loop system.<n>Our hardware experiments on quadruped locomotion demonstrate that ContractionPPO enables robust, certifiably stable control even under strong external perturbations.
• Score: 5.690649768462432
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Legged locomotion in unstructured environments demands not only high-performance control policies but also formal guarantees to ensure robustness under perturbations. Control methods often require carefully designed reference trajectories, which are challenging to construct in high-dimensional, contact-rich systems such as quadruped robots. In contrast, Reinforcement Learning (RL) directly learns policies that implicitly generate motion, and uniquely benefits from access to privileged information, such as full state and dynamics during training, that is not available at deployment. We present ContractionPPO, a framework for certified robust planning and control of legged robots by augmenting Proximal Policy Optimization (PPO) RL with a state-dependent contraction metric layer. This approach enables the policy to maximize performance while simultaneously producing a contraction metric that certifies incremental exponential stability of the simulated closed-loop system. The metric is parameterized as a Lipschitz neural network and trained jointly with the policy, either in parallel or as an auxiliary head of the PPO backbone. While the contraction metric is not deployed during real-world execution, we derive upper bounds on the worst-case contraction rate and show that these bounds ensure the learned contraction metric generalizes from simulation to real-world deployment. Our hardware experiments on quadruped locomotion demonstrate that ContractionPPO enables robust, certifiably stable control even under strong external perturbations.

Related papers

• PPO-EAL: Exact Augmented Lagrangian Proximal Policy Optimization for Safe Robotic Control [8.882504398064087]
  PPO-EAL is a novel first-order constrained policy optimization framework.<n>It integrates exact augmented Lagrangian optimization into policy optimization for safe robotic control.<n>It achieves theoretically grounded constraint enforcement without requiring impractically large penalty factors.
  arXiv  Detail & Related papers  (2026-06-26T08:59:38Z)
• Neuromorphic Reinforcement Learning for Quadruped Locomotion Control on Uneven Terrain [7.828170373014957]
  Local learning can replace global backpropagation graphs with updates driven by local neural states.<n>This work proposes an equilibrium-propagation (EP)-based proximal policy optimization (PPO) framework for uneven-terrain quadruped locomotion.
  arXiv  Detail & Related papers  (2026-05-10T15:16:07Z)
• Learning Reactive Dexterous Grasping via Hierarchical Task-Space RL Planning and Joint-Space QP Control [50.28263951510334]
  We propose a hybrid hierarchical control framework for reactive dexterous grasping.<n>The proposed framework explicitly decouples high-level spatial intent from low-level joint execution.<n>We extensively validate the proposed framework through a rigorous simulation-to-reality pipeline.
  arXiv  Detail & Related papers  (2026-05-05T04:49:38Z)
• Deep Reinforcement Learning for Robotic Manipulation under Distribution Shift with Bounded Extremum Seeking [42.278434352794676]
  Reinforcement learning has shown strong performance in robotic manipulation, but learned policies often degrade in performance when test conditions differ from the training distribution.<n>This limitation is especially important in contact-rich tasks such as pushing and pick-and-place, where changes in goals, contact conditions, or robot dynamics can drive the system out-of-distribution at inference time.<n>In this paper, we investigate a hybrid controller that combines reinforcement learning with bounded extremum seeking to improve robustness under such conditions.
  arXiv  Detail & Related papers  (2026-04-01T16:59:01Z)
• Task-Specified Compliance Bounds for Humanoids via Lipschitz-Constrained Policies [1.7446104539598897]
  We propose an anisotropic Lipschitz-constrained policy (ALCP) that maps a task-space stiffness upper bound to a state-dependent Lipschitz-style constraint on the policy Jacobian.<n>The resulting constraint is enforced during RL training via a hinge-squared spectral-norm penalty, preserving physical interpretability while enabling direction-dependent compliance.
  arXiv  Detail & Related papers  (2026-03-17T07:00:32Z)
• An Optimal Control Approach To Transformer Training [7.136933021609078]
  We develop a rigorous optimal control-theoretic approach to Transformer training that respects key structural constraints.<n>We show that lifting it to probability measures produces a fully-observed Markov decision process (MDP)<n>To train a Transformer, we propose a triply quantized training procedure for the lifted MDP by quantizing the state space, the space of probability measures, and the action space.
  arXiv  Detail & Related papers  (2026-03-10T12:17:48Z)
• DFPO: Scaling Value Modeling via Distributional Flow towards Robust and Generalizable LLM Post-Training [94.568675548967]
  Training reinforcement learning (RL) systems in real-world environments remains challenging due to noisy supervision and poor out-of-domain generalization.<n>Recent distributional RL methods improve robustness by modeling values with multiple quantile points, but they still learn each quantile independently as a scalar.<n>We propose DFPO, a robust distributional RL framework that models values as continuous flows across time steps.
  arXiv  Detail & Related papers  (2026-02-05T17:07:42Z)
• Rethinking the Trust Region in LLM Reinforcement Learning [72.25890308541334]
  Proximal Policy Optimization (PPO) serves as the de facto standard algorithm for Large Language Models (LLMs)<n>We propose Divergence Proximal Policy Optimization (DPPO), which substitutes clipping with a more principled constraint.<n>DPPO achieves superior training and efficiency compared to existing methods, offering a more robust foundation for RL-based fine-tuning.
  arXiv  Detail & Related papers  (2026-02-04T18:59:04Z)
• Relative Entropy Pathwise Policy Optimization [66.03329137921949]
  We present an on-policy algorithm that trains Q-value models purely from on-policy trajectories.<n>We show how to combine policies for exploration with constrained updates for stable training, and evaluate important architectural components that stabilize value function learning.
  arXiv  Detail & Related papers  (2025-07-15T06:24:07Z)
• Integrating DeepRL with Robust Low-Level Control in Robotic Manipulators for Non-Repetitive Reaching Tasks [0.24578723416255746]
  In robotics, contemporary strategies are learning-based, characterized by a complex black-box nature and a lack of interpretability. We propose integrating a collision-free trajectory planner based on deep reinforcement learning (DRL) with a novel auto-tuning low-level control strategy.
  arXiv  Detail & Related papers  (2024-02-04T15:54:03Z)
• Safety Correction from Baseline: Towards the Risk-aware Policy in Robotics via Dual-agent Reinforcement Learning [64.11013095004786]
  We propose a dual-agent safe reinforcement learning strategy consisting of a baseline and a safe agent. Such a decoupled framework enables high flexibility, data efficiency and risk-awareness for RL-based control. The proposed method outperforms the state-of-the-art safe RL algorithms on difficult robot locomotion and manipulation tasks.
  arXiv  Detail & Related papers  (2022-12-14T03:11:25Z)
• Modular Deep Reinforcement Learning for Continuous Motion Planning with Temporal Logic [59.94347858883343]
  This paper investigates the motion planning of autonomous dynamical systems modeled by Markov decision processes (MDP) The novelty is to design an embedded product MDP (EP-MDP) between the LDGBA and the MDP. The proposed LDGBA-based reward shaping and discounting schemes for the model-free reinforcement learning (RL) only depend on the EP-MDP states.
  arXiv  Detail & Related papers  (2021-02-24T01:11:25Z)
• Guided Constrained Policy Optimization for Dynamic Quadrupedal Robot Locomotion [78.46388769788405]
  We introduce guided constrained policy optimization (GCPO), an RL framework based upon our implementation of constrained policy optimization (CPPO) We show that guided constrained RL offers faster convergence close to the desired optimum resulting in an optimal, yet physically feasible, robotic control behavior without the need for precise reward function tuning.
  arXiv  Detail & Related papers  (2020-02-22T10:15:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.