Spectral Normalization for Lipschitz-Constrained Policies on Learning Humanoid Locomotion

• URL: http://arxiv.org/abs/2504.08246v1
• Date: Fri, 11 Apr 2025 04:12:15 GMT
• Title: Spectral Normalization for Lipschitz-Constrained Policies on Learning Humanoid Locomotion
• Abstract summary: This work proposes Spectral Normalization (SN) as an efficient replacement for enforcing Lipschitz continuity.<n> SN effectively limits high-frequency policy fluctuations while significantly reducing GPU memory usage.<n> Experimental evaluations in both simulation and real-world humanoid robot show that SN achieves performance comparable to gradient penalty methods.
• Score: 12.790605143952996
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Reinforcement learning (RL) has shown great potential in training agile and adaptable controllers for legged robots, enabling them to learn complex locomotion behaviors directly from experience. However, policies trained in simulation often fail to transfer to real-world robots due to unrealistic assumptions such as infinite actuator bandwidth and the absence of torque limits. These conditions allow policies to rely on abrupt, high-frequency torque changes, which are infeasible for real actuators with finite bandwidth. Traditional methods address this issue by penalizing aggressive motions through regularization rewards, such as joint velocities, accelerations, and energy consumption, but they require extensive hyperparameter tuning. Alternatively, Lipschitz-Constrained Policies (LCP) enforce finite bandwidth action control by penalizing policy gradients, but their reliance on gradient calculations introduces significant GPU memory overhead. To overcome this limitation, this work proposes Spectral Normalization (SN) as an efficient replacement for enforcing Lipschitz continuity. By constraining the spectral norm of network weights, SN effectively limits high-frequency policy fluctuations while significantly reducing GPU memory usage. Experimental evaluations in both simulation and real-world humanoid robot show that SN achieves performance comparable to gradient penalty methods while enabling more efficient parallel training.

Related papers

• Physics-Informed Reinforcement Learning of Spatial Density Velocity Potentials for Map-Free Racing [73.88859384645264]
  Out-Of-Distribution (OOD) generalization to various racetracks utilizes Machine Learning (ML) to encode the mathematical relation between sensor data and vehicle actuation for end-to-end control.<n>This paper presents a DRL method that parameterizes nonlinear vehicle dynamics from the spectral distribution of depth measurements with a non-geometric, physics-informed reward, to infer vehicle time-optimal and overtaking racing controls with an Artificial Neural Network (ANN)<n>The policy outperforms human demonstrations by 12% in OOD tracks on proportionally scaled hardware, by maximizing the friction circle with tire dynamics that resemble an empirical Pacejka tire model.
  arXiv  Detail & Related papers  (2026-04-10T17:12:07Z)
• FlashSAC: Fast and Stable Off-Policy Reinforcement Learning for High-Dimensional Robot Control [55.38832429564216]
  Reinforcement learning (RL) is a core approach for robot control when expert demonstrations are unavailable.<n>We present FlashSAC, a fast and stable off-policy RL algorithm built on Soft Actor-Critic.<n>Across over 60 tasks in 10 simulators, FlashSAC consistently outperforms PPO and strong off-policy baselines in both final performance and training efficiency.
  arXiv  Detail & Related papers  (2026-04-06T09:03:41Z)
• ContractionPPO: Certified Reinforcement Learning via Differentiable Contraction Layers [5.690649768462432]
  We present ContractionPPO, a framework for robust planning and control of legged robots.<n>ContractionPPO produces a contraction metric that certifies exponential stability of the simulated closed-loop system.<n>Our hardware experiments on quadruped locomotion demonstrate that ContractionPPO enables robust, certifiably stable control even under strong external perturbations.
  arXiv  Detail & Related papers  (2026-03-20T04:32:18Z)
• Task-Specified Compliance Bounds for Humanoids via Lipschitz-Constrained Policies [1.7446104539598897]
  We propose an anisotropic Lipschitz-constrained policy (ALCP) that maps a task-space stiffness upper bound to a state-dependent Lipschitz-style constraint on the policy Jacobian.<n>The resulting constraint is enforced during RL training via a hinge-squared spectral-norm penalty, preserving physical interpretability while enabling direction-dependent compliance.
  arXiv  Detail & Related papers  (2026-03-17T07:00:32Z)
• Learning Smooth Time-Varying Linear Policies with an Action Jacobian Penalty [1.8122712065585906]
  Reinforcement learning provides a framework for learning control policies that can reproduce diverse motions for simulated characters.<n>Existing work addresses this issue by adding a reward term that penalizes a large change in actions over time.<n>We propose to use the action Jacobian penalty, which penalizes changes in action with respect to the changes in simulated state directly through auto differentiation.
  arXiv  Detail & Related papers  (2026-02-20T16:11:19Z)
• Learning Smooth Humanoid Locomotion through Lipschitz-Constrained Policies [37.124201522993076]
  Lipschitz-Constrained Policies (LCP) is a method that imposes a Lipschitz constraint on a learned policy. We show that LCP effectively replaces the need for smoothing rewards or low-pass filters. We evaluate LCP in both simulation and real-world humanoid robots, producing smooth and robust locomotion controllers.
  arXiv  Detail & Related papers  (2024-10-15T17:52:20Z)
• Autonomous Vehicle Controllers From End-to-End Differentiable Simulation [60.05963742334746]
  We propose a differentiable simulator and design an analytic policy gradients (APG) approach to training AV controllers. Our proposed framework brings the differentiable simulator into an end-to-end training loop, where gradients of environment dynamics serve as a useful prior to help the agent learn a more grounded policy. We find significant improvements in performance and robustness to noise in the dynamics, as well as overall more intuitive human-like handling.
  arXiv  Detail & Related papers  (2024-09-12T11:50:06Z)
• MOSEAC: Streamlined Variable Time Step Reinforcement Learning [14.838483990647697]
  We introduce the Multi-Objective Soft Elastic Actor-Critic (MOSEAC) method. MOSEAC features an adaptive reward scheme based on observed trends in task rewards during training. We validate the MOSEAC method through simulations in a Newtonian kinematics environment.
  arXiv  Detail & Related papers  (2024-06-03T16:51:57Z)
• Efficiently Training Deep-Learning Parametric Policies using Lagrangian Duality [55.06411438416805]
  Constrained Markov Decision Processes (CMDPs) are critical in many high-stakes applications.<n>This paper introduces a novel approach, Two-Stage Deep Decision Rules (TS- DDR) to efficiently train parametric actor policies.<n>It is shown to enhance solution quality and to reduce computation times by several orders of magnitude when compared to current state-of-the-art methods.
  arXiv  Detail & Related papers  (2024-05-23T18:19:47Z)
• Distilling Reinforcement Learning Policies for Interpretable Robot Locomotion: Gradient Boosting Machines and Symbolic Regression [53.33734159983431]
  This paper introduces a novel approach to distill neural RL policies into more interpretable forms. We train expert neural network policies using RL and distill them into (i) GBMs, (ii) EBMs, and (iii) symbolic policies.
  arXiv  Detail & Related papers  (2024-03-21T11:54:45Z)
• Reinforcement Learning with Elastic Time Steps [14.838483990647697]
  Multi-Objective Soft Elastic Actor-Critic (MOSEAC) is an off-policy actor-critic algorithm that uses elastic time steps to dynamically adjust the control frequency. We show that MOSEAC converges and produces stable policies at the theoretical level, and validate our findings in a real-time 3D racing game.
  arXiv  Detail & Related papers  (2024-02-22T20:49:04Z)
• Action-Quantized Offline Reinforcement Learning for Robotic Skill Learning [68.16998247593209]
  offline reinforcement learning (RL) paradigm provides recipe to convert static behavior datasets into policies that can perform better than the policy that collected the data. In this paper, we propose an adaptive scheme for action quantization. We show that several state-of-the-art offline RL methods such as IQL, CQL, and BRAC improve in performance on benchmarks when combined with our proposed discretization scheme.
  arXiv  Detail & Related papers  (2023-10-18T06:07:10Z)
• Accelerated Policy Learning with Parallel Differentiable Simulation [59.665651562534755]
  We present a differentiable simulator and a new policy learning algorithm (SHAC) Our algorithm alleviates problems with local minima through a smooth critic function. We show substantial improvements in sample efficiency and wall-clock time over state-of-the-art RL and differentiable simulation-based algorithms.
  arXiv  Detail & Related papers  (2022-04-14T17:46:26Z)
• Guided Constrained Policy Optimization for Dynamic Quadrupedal Robot Locomotion [78.46388769788405]
  We introduce guided constrained policy optimization (GCPO), an RL framework based upon our implementation of constrained policy optimization (CPPO) We show that guided constrained RL offers faster convergence close to the desired optimum resulting in an optimal, yet physically feasible, robotic control behavior without the need for precise reward function tuning.
  arXiv  Detail & Related papers  (2020-02-22T10:15:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.