Mobile-TeleVision: Predictive Motion Priors for Humanoid Whole-Body Control

• URL: http://arxiv.org/abs/2412.07773v1
• Date: Tue, 10 Dec 2024 18:59:50 GMT
• Title: Mobile-TeleVision: Predictive Motion Priors for Humanoid Whole-Body Control
• Authors: Chenhao Lu, Xuxin Cheng, Jialong Li, Shiqi Yang, Mazeyu Ji, Chengjing Yuan, Ge Yang, Sha Yi, Xiaolong Wang,
• Abstract summary: Humanoid robots require robust lower-body locomotion and precise upper-body manipulation. Recent Reinforcement Learning approaches provide whole-body loco-manipulation policies, but lack precise manipulation. We introduce high-body kinematic control using inverses (IK) and motion for precise manipulation. We show that CVAE features are crucial for stability and robustness, and significantly outperforms RL-based whole-body control in precise manipulation.
• Score: 18.269588421166503
• License:
• Abstract: Humanoid robots require both robust lower-body locomotion and precise upper-body manipulation. While recent Reinforcement Learning (RL) approaches provide whole-body loco-manipulation policies, they lack precise manipulation with high DoF arms. In this paper, we propose decoupling upper-body control from locomotion, using inverse kinematics (IK) and motion retargeting for precise manipulation, while RL focuses on robust lower-body locomotion. We introduce PMP (Predictive Motion Priors), trained with Conditional Variational Autoencoder (CVAE) to effectively represent upper-body motions. The locomotion policy is trained conditioned on this upper-body motion representation, ensuring that the system remains robust with both manipulation and locomotion. We show that CVAE features are crucial for stability and robustness, and significantly outperforms RL-based whole-body control in precise manipulation. With precise upper-body motion and robust lower-body locomotion control, operators can remotely control the humanoid to walk around and explore different environments, while performing diverse manipulation tasks.

Related papers

• HOMIE: Humanoid Loco-Manipulation with Isomorphic Exoskeleton Cockpit [52.12750762494588]
  Current humanoid teleoperation systems either lack reliable low-level control policies, or struggle to acquire accurate whole-body control commands. We propose a novel humanoid teleoperation cockpit integrates a humanoid loco-manipulation policy and a low-cost exoskeleton-based hardware system.
  arXiv  Detail & Related papers  (2025-02-18T16:33:38Z)
• A Unified and General Humanoid Whole-Body Controller for Fine-Grained Locomotion [30.418274871034775]
  We propose HugWBC: a unified and general humanoid whole-body controller for fine-grained locomotion. HuGWBC enables real-world humanoid robots to produce various natural gaits, including walking (running), jumping, standing, and hopping, with customizable parameters. HuGWBC also supports real-time interventions from external upper-body controllers like teleoperation, enabling loco-manipulation.
  arXiv  Detail & Related papers  (2025-02-05T14:26:01Z)
• Learning Force Control for Legged Manipulation [18.894304288225385]
  We propose a method for training RL policies for direct force control without requiring access to force sensing. We showcase our method on a whole-body control platform of a quadruped robot with an arm. We provide the first deployment of learned whole-body force control in legged manipulators, paving the way for more versatile and adaptable legged robots.
  arXiv  Detail & Related papers  (2024-05-02T15:53:43Z)
• Visual Whole-Body Control for Legged Loco-Manipulation [22.50054654508986]
  We study the problem of mobile manipulation using legged robots equipped with an arm. We propose a framework that can conduct the whole-body control autonomously with visual observations.
  arXiv  Detail & Related papers  (2024-03-25T17:26:08Z)
• Reinforcement Learning for Versatile, Dynamic, and Robust Bipedal Locomotion Control [106.32794844077534]
  This paper presents a study on using deep reinforcement learning to create dynamic locomotion controllers for bipedal robots. We develop a general control solution that can be used for a range of dynamic bipedal skills, from periodic walking and running to aperiodic jumping and standing. This work pushes the limits of agility for bipedal robots through extensive real-world experiments.
  arXiv  Detail & Related papers  (2024-01-30T10:48:43Z)
• TLControl: Trajectory and Language Control for Human Motion Synthesis [68.09806223962323]
  We present TLControl, a novel method for realistic human motion synthesis. It incorporates both low-level Trajectory and high-level Language semantics controls. It is practical for interactive and high-quality animation generation.
  arXiv  Detail & Related papers  (2023-11-28T18:54:16Z)
• Universal Humanoid Motion Representations for Physics-Based Control [71.46142106079292]
  We present a universal motion representation that encompasses a comprehensive range of motor skills for physics-based humanoid control. We first learn a motion imitator that can imitate all of human motion from a large, unstructured motion dataset. We then create our motion representation by distilling skills directly from the imitator.
  arXiv  Detail & Related papers  (2023-10-06T20:48:43Z)
• QuestSim: Human Motion Tracking from Sparse Sensors with Simulated Avatars [80.05743236282564]
  Real-time tracking of human body motion is crucial for immersive experiences in AR/VR. We present a reinforcement learning framework that takes in sparse signals from an HMD and two controllers. We show that a single policy can be robust to diverse locomotion styles, different body sizes, and novel environments.
  arXiv  Detail & Related papers  (2022-09-20T00:25:54Z)
• Agile Maneuvers in Legged Robots: a Predictive Control Approach [20.55884151818753]
  We present a contact-phase predictive and state-feedback controllers that enables legged robots to plan and perform agile locomotion skills. Our work is the first to show that predictive control can handle actuation limits, generate agile locomotion maneuvers and execute locally optimal feedback policies on hardware without the use of a separate whole-body controller.
  arXiv  Detail & Related papers  (2022-03-14T23:32:17Z)
• Reinforcement Learning for Robust Parameterized Locomotion Control of Bipedal Robots [121.42930679076574]
  We present a model-free reinforcement learning framework for training robust locomotion policies in simulation. domain randomization is used to encourage the policies to learn behaviors that are robust across variations in system dynamics. We demonstrate this on versatile walking behaviors such as tracking a target walking velocity, walking height, and turning yaw.
  arXiv  Detail & Related papers  (2021-03-26T07:14:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.