Learning Multi-Modal Whole-Body Control for Real-World Humanoid Robots

• URL: http://arxiv.org/abs/2408.07295v2
• Date: Mon, 16 Sep 2024 19:41:39 GMT
• Title: Learning Multi-Modal Whole-Body Control for Real-World Humanoid Robots
• Authors: Pranay Dugar, Aayam Shrestha, Fangzhou Yu, Bart van Marum, Alan Fern,
• Abstract summary: Masked Humanoid Controller (MHC) supports standing, walking, and mimicry of whole and partial-body motions. MHC imitates partially masked motions from a library of behaviors spanning standing, walking, optimized reference trajectories, re-targeted video clips, and human motion capture data. We demonstrate sim-to-real transfer on the real-world Digit V3 humanoid robot.
• Score: 13.229028132036321
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The foundational capabilities of humanoid robots should include robustly standing, walking, and mimicry of whole and partial-body motions. This work introduces the Masked Humanoid Controller (MHC), which supports all of these capabilities by tracking target trajectories over selected subsets of humanoid state variables while ensuring balance and robustness against disturbances. The MHC is trained in simulation using a carefully designed curriculum that imitates partially masked motions from a library of behaviors spanning standing, walking, optimized reference trajectories, re-targeted video clips, and human motion capture data. It also allows for combining joystick-based control with partial-body motion mimicry. We showcase simulation experiments validating the MHC's ability to execute a wide variety of behaviors from partially-specified target motions. Moreover, we demonstrate sim-to-real transfer on the real-world Digit V3 humanoid robot. To our knowledge, this is the first instance of a learned controller that can realize whole-body control of a real-world humanoid for such diverse multi-modal targets.

Related papers

• HOMIE: Humanoid Loco-Manipulation with Isomorphic Exoskeleton Cockpit [52.12750762494588]
  Current humanoid teleoperation systems either lack reliable low-level control policies, or struggle to acquire accurate whole-body control commands. We propose a novel humanoid teleoperation cockpit integrates a humanoid loco-manipulation policy and a low-cost exoskeleton-based hardware system.
  arXiv  Detail & Related papers  (2025-02-18T16:33:38Z)
• Learning Humanoid Standing-up Control across Diverse Postures [27.79222176982376]
  We present HoST (Humanoid Standing-up Control), a reinforcement learning framework that learns standing-up control from scratch. HoST effectively learns posture-adaptive motions by leveraging a multi-critic architecture and curriculum-based training on diverse simulated terrains. Our experimental results demonstrate that the controllers achieve smooth, stable, and robust standing-up motions across a wide range of laboratory and outdoor environments.
  arXiv  Detail & Related papers  (2025-02-12T13:10:09Z)
• Generating Physically Realistic and Directable Human Motions from Multi-Modal Inputs [16.41735119504929]
  This work focuses on generating realistic, physically-based human behaviors from multi-modal inputs, which may only partially specify the desired motion. The input may come from a VR controller providing arm motion and body velocity, partial key-point animation, computer vision applied to videos, or even higher-level motion goals. We introduce the Masked Humanoid Controller (MHC), a novel approach that applies multi-objective imitation learning on augmented and selectively masked motion demonstrations.
  arXiv  Detail & Related papers  (2025-02-08T17:02:11Z)
• ExBody2: Advanced Expressive Humanoid Whole-Body Control [16.69009772546575]
  We propose ExBody2, a whole-body tracking framework that can control the humanoid to mimic the motion. The model is trained in simulation with Reinforcement Learning and then transferred to the real world. We conduct experiments on two humanoid platforms and demonstrate the superiority of our approach against state-of-the-arts.
  arXiv  Detail & Related papers  (2024-12-17T18:59:51Z)
• Learning Human-to-Humanoid Real-Time Whole-Body Teleoperation [34.65637397405485]
  We present Human to Humanoid (H2O), a framework that enables real-time whole-body teleoperation of a humanoid robot with only an RGB camera. We train a robust real-time humanoid motion imitator in simulation using these refined motions and transfer it to the real humanoid robot in a zero-shot manner. To the best of our knowledge, this is the first demonstration to achieve learning-based real-time whole-body humanoid teleoperation.
  arXiv  Detail & Related papers  (2024-03-07T12:10:41Z)
• Expressive Whole-Body Control for Humanoid Robots [20.132927075816742]
  We learn a whole-body control policy on a human-sized robot to mimic human motions as realistic as possible. With training in simulation and Sim2Real transfer, our policy can control a humanoid robot to walk in different styles, shake hands with humans, and even dance with a human in the real world.
  arXiv  Detail & Related papers  (2024-02-26T18:09:24Z)
• Universal Humanoid Motion Representations for Physics-Based Control [71.46142106079292]
  We present a universal motion representation that encompasses a comprehensive range of motor skills for physics-based humanoid control. We first learn a motion imitator that can imitate all of human motion from a large, unstructured motion dataset. We then create our motion representation by distilling skills directly from the imitator.
  arXiv  Detail & Related papers  (2023-10-06T20:48:43Z)
• Perpetual Humanoid Control for Real-time Simulated Avatars [77.05287269685911]
  We present a physics-based humanoid controller that achieves high-fidelity motion imitation and fault-tolerant behavior. Our controller scales up to learning ten thousand motion clips without using any external stabilizing forces. We demonstrate the effectiveness of our controller by using it to imitate noisy poses from video-based pose estimators and language-based motion generators in a live and real-time multi-person avatar use case.
  arXiv  Detail & Related papers  (2023-05-10T20:51:37Z)
• Real-World Humanoid Locomotion with Reinforcement Learning [92.85934954371099]
  We present a fully learning-based approach for real-world humanoid locomotion. Our controller can walk over various outdoor terrains, is robust to external disturbances, and can adapt in context.
  arXiv  Detail & Related papers  (2023-03-06T18:59:09Z)
• QuestSim: Human Motion Tracking from Sparse Sensors with Simulated Avatars [80.05743236282564]
  Real-time tracking of human body motion is crucial for immersive experiences in AR/VR. We present a reinforcement learning framework that takes in sparse signals from an HMD and two controllers. We show that a single policy can be robust to diverse locomotion styles, different body sizes, and novel environments.
  arXiv  Detail & Related papers  (2022-09-20T00:25:54Z)
• Visual Navigation Among Humans with Optimal Control as a Supervisor [72.5188978268463]
  We propose an approach that combines learning-based perception with model-based optimal control to navigate among humans. Our approach is enabled by our novel data-generation tool, HumANav. We demonstrate that the learned navigation policies can anticipate and react to humans without explicitly predicting future human motion.
  arXiv  Detail & Related papers  (2020-03-20T16:13:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.