Learning Smooth Humanoid Locomotion through Lipschitz-Constrained Policies

• URL: http://arxiv.org/abs/2410.11825v3
• Date: Mon, 28 Oct 2024 09:46:19 GMT
• Title: Learning Smooth Humanoid Locomotion through Lipschitz-Constrained Policies
• Abstract summary: Lipschitz-Constrained Policies (LCP) is a method that imposes a Lipschitz constraint on a learned policy. We show that LCP effectively replaces the need for smoothing rewards or low-pass filters. We evaluate LCP in both simulation and real-world humanoid robots, producing smooth and robust locomotion controllers.
• Score: 37.124201522993076
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Reinforcement learning combined with sim-to-real transfer offers a general framework for developing locomotion controllers for legged robots. To facilitate successful deployment in the real world, smoothing techniques, such as low-pass filters and smoothness rewards, are often employed to develop policies with smooth behaviors. However, because these techniques are non-differentiable and usually require tedious tuning of a large set of hyperparameters, they tend to require extensive manual tuning for each robotic platform. To address this challenge and establish a general technique for enforcing smooth behaviors, we propose a simple and effective method that imposes a Lipschitz constraint on a learned policy, which we refer to as Lipschitz-Constrained Policies (LCP). We show that the Lipschitz constraint can be implemented in the form of a gradient penalty, which provides a differentiable objective that can be easily incorporated with automatic differentiation frameworks. We demonstrate that LCP effectively replaces the need for smoothing rewards or low-pass filters and can be easily integrated into training frameworks for many distinct humanoid robots. We extensively evaluate LCP in both simulation and real-world humanoid robots, producing smooth and robust locomotion controllers. All simulation and deployment code, along with complete checkpoints, is available on our project page: https://lipschitz-constrained-policy.github.io.

Related papers

• Actuator Reality Shaping for Zero-Shot Sim-to-Real Robot Learning [0.5277024349608834]
  Sim-to-real transfer in robot learning is often limited by discrepancies between the ideal actuator dynamics assumed during policy training and the nonlinear, hardware-dependent behavior of physical motors.<n>We introduce an alternative paradigm: actuator reality shaping.<n>We validate the approach on a single-joint high-gear-ratio servo under external loads and a 7-DOF robotic arm reaching task.
  arXiv  Detail & Related papers  (2026-07-02T14:12:41Z)
• Mind Your Steps: A General Learning Framework for Accurate Humanoid Foothold Tracking [78.30338456206984]
  We introduce a novel framework for training general-purpose 3D foothold-tracking policies.<n>By dynamically providing footstep support through a goal sampler, this method enables the learned policy to be agnostic to specific terrains.<n>Our new target representation effectively mitigates challenges arising in the real world, such as noisy and inaccurate pose estimation and foot contact estimation.
  arXiv  Detail & Related papers  (2026-06-06T16:44:56Z)
• X-OP: Cross-Morphology Whole-Body Teleoperation via MPC Retargeting [3.134050607080983]
  Whole-body teleoperation is essential for scalable robot data collection in loco-manipulation tasks.<n>We propose a hierarchical whole-body teleoperation framework driven by a single XR device.<n>We show higher success rates on whole-body control tasks for both a humanoid and a mobile manipulator.
  arXiv  Detail & Related papers  (2026-06-06T01:50:59Z)
• Sim-to-Real Transfer for Muscle-Actuated Robots via Generalized Actuator Networks [53.12548325396181]
  Tendon drives paired with soft muscle actuation enable faster and safer robots.<n>We propose a sim-to-real pipeline that learns a neural network model of this complex actuation.<n>We successfully deploy precise goal-reaching and dynamic ball-in-a-cup policies trained entirely in simulation.
  arXiv  Detail & Related papers  (2026-04-10T16:52:54Z)
• Flow Policy Gradients for Robot Control [67.61978635211048]
  Flow matching policy gradients can be made effective for training and fine-tuning more expressive policies.<n>We show how policies can exploit the flow representation for exploration when training from scratch, as well as improved fine-tuning robustness over baselines.
  arXiv  Detail & Related papers  (2026-02-02T18:56:49Z)
• SPACeR: Self-Play Anchoring with Centralized Reference Models [50.55045557371374]
  Sim agent policies are realistic, human-like, fast, and scalable in multi-agent settings.<n>Recent progress in imitation learning with large diffusion-based or tokenized models has shown that behaviors can be captured directly from human driving data.<n>We propose SPACeR, a framework that leverages a pretrained tokenized autoregressive motion model as a central reference policy.
  arXiv  Detail & Related papers  (2025-10-20T19:53:02Z)
• Spectral Normalization for Lipschitz-Constrained Policies on Learning Humanoid Locomotion [12.790605143952996]
  This work proposes Spectral Normalization (SN) as an efficient replacement for enforcing Lipschitz continuity. SN effectively limits high-frequency policy fluctuations while significantly reducing GPU memory usage. Experimental evaluations in both simulation and real-world humanoid robot show that SN achieves performance comparable to gradient penalty methods.
  arXiv  Detail & Related papers  (2025-04-11T04:12:15Z)
• Autonomous Vehicle Controllers From End-to-End Differentiable Simulation [60.05963742334746]
  We propose a differentiable simulator and design an analytic policy gradients (APG) approach to training AV controllers. Our proposed framework brings the differentiable simulator into an end-to-end training loop, where gradients of environment dynamics serve as a useful prior to help the agent learn a more grounded policy. We find significant improvements in performance and robustness to noise in the dynamics, as well as overall more intuitive human-like handling.
  arXiv  Detail & Related papers  (2024-09-12T11:50:06Z)
• Distilling Reinforcement Learning Policies for Interpretable Robot Locomotion: Gradient Boosting Machines and Symbolic Regression [53.33734159983431]
  This paper introduces a novel approach to distill neural RL policies into more interpretable forms. We train expert neural network policies using RL and distill them into (i) GBMs, (ii) EBMs, and (iii) symbolic policies.
  arXiv  Detail & Related papers  (2024-03-21T11:54:45Z)
• Differentiable Constrained Imitation Learning for Robot Motion Planning and Control [0.26999000177990923]
  We develop a framework for constraint robotic motion planning and control, as well as traffic agent simulation. We focus on mobile robot and automated driving applications. Simulated experiments of mobile robot navigation and automated driving provide evidence for the performance of the proposed method.
  arXiv  Detail & Related papers  (2022-10-21T08:19:45Z)
• Using Simulation Optimization to Improve Zero-shot Policy Transfer of Quadrotors [0.14999444543328289]
  We show that it is possible to train low-level control policies with reinforcement learning entirely in simulation and deploy them on a quadrotor robot without using real-world data to fine-tune. Our neural network-based policies use only onboard sensor data and run entirely on the embedded drone hardware.
  arXiv  Detail & Related papers  (2022-01-04T22:32:05Z)
• Teaching a Robot to Walk Using Reinforcement Learning [0.0]
  reinforcement learning can train optimal walking policies with ease. We teach a simulated two-dimensional bipedal robot how to walk using the OpenAI Gym BipedalWalker-v3 environment. ARS resulted in a better trained robot, and produced an optimal policy which officially "solves" the BipedalWalker-v3 problem.
  arXiv  Detail & Related papers  (2021-12-13T21:35:45Z)
• Zero-Shot Uncertainty-Aware Deployment of Simulation Trained Policies on Real-World Robots [17.710172337571617]
  Deep reinforcement learning (RL) agents tend to make errors when deployed in the real world due to mismatches between the training and execution environments. We propose a novel uncertainty-aware deployment strategy that combines the strengths of deep RL policies and traditional handcrafted controllers. We show promising results on two real-world continuous control tasks, where BCF outperforms both the standalone policy and controller.
  arXiv  Detail & Related papers  (2021-12-10T02:13:01Z)
• Composable Learning with Sparse Kernel Representations [110.19179439773578]
  We present a reinforcement learning algorithm for learning sparse non-parametric controllers in a Reproducing Kernel Hilbert Space. We improve the sample complexity of this approach by imposing a structure of the state-action function through a normalized advantage function. We demonstrate the performance of this algorithm on learning obstacle-avoidance policies in multiple simulations of a robot equipped with a laser scanner while navigating in a 2D environment.
  arXiv  Detail & Related papers  (2021-03-26T13:58:23Z)
• Learning from Simulation, Racing in Reality [126.56346065780895]
  We present a reinforcement learning-based solution to autonomously race on a miniature race car platform. We show that a policy that is trained purely in simulation can be successfully transferred to the real robotic setup.
  arXiv  Detail & Related papers  (2020-11-26T14:58:49Z)
• Guided Constrained Policy Optimization for Dynamic Quadrupedal Robot Locomotion [78.46388769788405]
  We introduce guided constrained policy optimization (GCPO), an RL framework based upon our implementation of constrained policy optimization (CPPO) We show that guided constrained RL offers faster convergence close to the desired optimum resulting in an optimal, yet physically feasible, robotic control behavior without the need for precise reward function tuning.
  arXiv  Detail & Related papers  (2020-02-22T10:15:53Z)
• Sim2Real Transfer for Reinforcement Learning without Dynamics Randomization [0.0]
  We show how to use the Operational Space Control framework (OSC) under joint and cartesian constraints for reinforcement learning in cartesian space. Our method is able to learn fast and with adjustable degrees of freedom, while we are able to transfer policies without additional dynamics randomizations.
  arXiv  Detail & Related papers  (2020-02-19T11:10:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.