Physics-Informed Policy Optimization via Analytic Dynamics Regularization

• URL: http://arxiv.org/abs/2603.14469v2
• Date: Sat, 21 Mar 2026 06:26:58 GMT
• Title: Physics-Informed Policy Optimization via Analytic Dynamics Regularization
• Abstract summary: Reinforcement learning (RL) has achieved strong performance in robotic control.<n>State-of-the-art policy learning methods, such as actor-critic methods, still suffer from high sample complexity.<n>We introduce a novel physics-informed RL framework, called PIPER, that seamlessly integrates physical constraints directly into neural policy optimization.
• Score: 5.537513058221965
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reinforcement learning (RL) has achieved strong performance in robotic control; however, state-of-the-art policy learning methods, such as actor-critic methods, still suffer from high sample complexity and often produce physically inconsistent actions. This limitation stems from neural policies implicitly rediscovering complex physics from data alone, despite accurate dynamics models being readily available in simulators. In this paper, we introduce a novel physics-informed RL framework, called PIPER, that seamlessly integrates physical constraints directly into neural policy optimization with analytical soft physics constraints. At the core of our method is the integration of a differentiable Lagrangian residual as a regularization term within the actor's objective. This residual, extracted from a robot's simulator description, subtly biases policy updates towards dynamically consistent solutions. Crucially, this physics integration is realized through an additional loss term during policy optimization, requiring no alterations to existing simulators or core RL algorithms. Extensive experiments demonstrate that our method significantly improves learning efficiency, stability, and control accuracy, establishing a new paradigm for efficient and physically consistent robotic control.

Related papers

• Continual Robot Policy Learning via Variational Neural Dynamics [20.3323041710497]
  Most learning-based controllers are trained once and deployed as if learning were complete.<n>This prevents the robot from using deployment experience to further improve task performance.<n>We propose a continual learning framework that uses real-world experience to improve robot policies under hidden and recurring dynamics.
  arXiv  Detail & Related papers  (2026-06-25T17:56:04Z)
• Dynamic Manipulation of Deformable Objects in 3D: Simulation, Benchmark and Learning Strategy [88.8665000676562]
  Prior methods often simplify the problem to low-speed or 2D settings, limiting their applicability to real-world 3D tasks.<n>To mitigate data scarcity, we introduce a novel simulation framework and benchmark grounded in reduced-order dynamics.<n>We propose Dynamics Informed Diffusion Policy (DIDP), a framework that integrates imitation pretraining with physics-informed test-time adaptation.
  arXiv  Detail & Related papers  (2025-05-23T03:28:25Z)
• Offline Robotic World Model: Learning Robotic Policies without a Physics Simulator [50.191655141020505]
  Reinforcement Learning (RL) has demonstrated impressive capabilities in robotic control but remains challenging due to high sample complexity, safety concerns, and the sim-to-real gap.<n>We introduce Offline Robotic World Model (RWM-O), a model-based approach that explicitly estimates uncertainty to improve policy learning without reliance on a physics simulator.
  arXiv  Detail & Related papers  (2025-04-23T12:58:15Z)
• Stochastic Online Optimization for Cyber-Physical and Robotic Systems [9.392372266209103]
  We propose a novel online framework for solving programming problems in the context of cyber-physical and robotic systems. Our problem formulation constraints model the evolution of a cyber-physical system, which has, in general, a continuous state and action space space is nonlinear. We show that even rough estimates of the dynamics can significantly improve the convergence of our algorithms.
  arXiv  Detail & Related papers  (2024-04-08T09:08:59Z)
• Imitation Learning via Differentiable Physics [26.356669151969953]
  Imitation learning (IL) methods such as inverse reinforcement learning (IRL) usually have a double-loop training process. We propose a new IL method, i.e., Imitation Learning via Differentiable Physics (ILD), which gets rid of the double-loop design. ILD achieves significant improvements in final performance, convergence speed, and stability.
  arXiv  Detail & Related papers  (2022-06-10T04:54:32Z)
• Gradient-Based Trajectory Optimization With Learned Dynamics [80.41791191022139]
  We use machine learning techniques to learn a differentiable dynamics model of the system from data. We show that a neural network can model highly nonlinear behaviors accurately for large time horizons. In our hardware experiments, we demonstrate that our learned model can represent complex dynamics for both the Spot and Radio-controlled (RC) car.
  arXiv  Detail & Related papers  (2022-04-09T22:07:34Z)
• Learning Robust Policy against Disturbance in Transition Dynamics via State-Conservative Policy Optimization [63.75188254377202]
  Deep reinforcement learning algorithms can perform poorly in real-world tasks due to discrepancy between source and target environments. We propose a novel model-free actor-critic algorithm to learn robust policies without modeling the disturbance in advance. Experiments in several robot control tasks demonstrate that SCPO learns robust policies against the disturbance in transition dynamics.
  arXiv  Detail & Related papers  (2021-12-20T13:13:05Z)
• PlasticineLab: A Soft-Body Manipulation Benchmark with Differentiable Physics [89.81550748680245]
  We introduce a new differentiable physics benchmark called PasticineLab. In each task, the agent uses manipulators to deform the plasticine into the desired configuration. We evaluate several existing reinforcement learning (RL) methods and gradient-based methods on this benchmark.
  arXiv  Detail & Related papers  (2021-04-07T17:59:23Z)
• Guided Constrained Policy Optimization for Dynamic Quadrupedal Robot Locomotion [78.46388769788405]
  We introduce guided constrained policy optimization (GCPO), an RL framework based upon our implementation of constrained policy optimization (CPPO) We show that guided constrained RL offers faster convergence close to the desired optimum resulting in an optimal, yet physically feasible, robotic control behavior without the need for precise reward function tuning.
  arXiv  Detail & Related papers  (2020-02-22T10:15:53Z)
• Automatic Differentiation and Continuous Sensitivity Analysis of Rigid Body Dynamics [15.565726546970678]
  We introduce a differentiable physics simulator for rigid body dynamics. In the context of trajectory optimization, we introduce a closed-loop model-predictive control algorithm.
  arXiv  Detail & Related papers  (2020-01-22T03:54:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.