H-GAP: Humanoid Control with a Generalist Planner

• URL: http://arxiv.org/abs/2312.02682v1
• Date: Tue, 5 Dec 2023 11:40:24 GMT
• Title: H-GAP: Humanoid Control with a Generalist Planner
• Authors: Zhengyao Jiang, Yingchen Xu, Nolan Wagener, Yicheng Luo, Michael Janner, Edward Grefenstette, Tim Rockt\"aschel, Yuandong Tian
• Abstract summary: Humanoid Generalist Autoencoding Planner (H-GAP) is a generative model trained on humanoid trajectories derived from human motioncaptured data. For 56 degrees of freedom humanoid, we empirically demonstrate that H-GAP learns to represent and generate a wide range of motor behaviours. We also do a series of empirical studies on the scaling properties of H-GAP, showing the potential for performance gains via additional data but not computing.
• Score: 45.50995825122686
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Humanoid control is an important research challenge offering avenues for integration into human-centric infrastructures and enabling physics-driven humanoid animations. The daunting challenges in this field stem from the difficulty of optimizing in high-dimensional action spaces and the instability introduced by the bipedal morphology of humanoids. However, the extensive collection of human motion-captured data and the derived datasets of humanoid trajectories, such as MoCapAct, paves the way to tackle these challenges. In this context, we present Humanoid Generalist Autoencoding Planner (H-GAP), a state-action trajectory generative model trained on humanoid trajectories derived from human motion-captured data, capable of adeptly handling downstream control tasks with Model Predictive Control (MPC). For 56 degrees of freedom humanoid, we empirically demonstrate that H-GAP learns to represent and generate a wide range of motor behaviours. Further, without any learning from online interactions, it can also flexibly transfer these behaviors to solve novel downstream control tasks via planning. Notably, H-GAP excels established MPC baselines that have access to the ground truth dynamics model, and is superior or comparable to offline RL methods trained for individual tasks. Finally, we do a series of empirical studies on the scaling properties of H-GAP, showing the potential for performance gains via additional data but not computing. Code and videos are available at https://ycxuyingchen.github.io/hgap/.

Related papers

• The Role of Domain Randomization in Training Diffusion Policies for Whole-Body Humanoid Control [14.36344580057985]
  Policies Diffusion (DPs) have shown impressive results in robotic manipulation. This paper investigates how dataset diversity and size affect the performance of DPs for humanoid whole-body control.
  arXiv  Detail & Related papers  (2024-11-02T19:33:28Z)
• ImDy: Human Inverse Dynamics from Imitated Observations [47.994797555884325]
  Inverse dynamics (ID) aims at reproducing the driven torques from human kinematic observations. Conventional optimization-based ID requires expensive laboratory setups, restricting its availability. We propose to exploit the recently progressive human motion imitation algorithms to learn human inverse dynamics in a data-driven manner.
  arXiv  Detail & Related papers  (2024-10-23T07:06:08Z)
• HINT: Learning Complete Human Neural Representations from Limited Viewpoints [69.76947323932107]
  We propose a NeRF-based algorithm able to learn a detailed and complete human model from limited viewing angles. As a result, our method can reconstruct complete humans even from a few viewing angles, increasing performance by more than 15% PSNR.
  arXiv  Detail & Related papers  (2024-05-30T05:43:09Z)
• Humanoid Locomotion as Next Token Prediction [84.21335675130021]
  Our model is a causal transformer trained via autoregressive prediction of sensorimotor trajectories. We show that our model enables a full-sized humanoid to walk in San Francisco zero-shot. Our model can transfer to the real world even when trained on only 27 hours of walking data, and can generalize commands not seen during training like walking backward.
  arXiv  Detail & Related papers  (2024-02-29T18:57:37Z)
• Learning Human Action Recognition Representations Without Real Humans [66.61527869763819]
  We present a benchmark that leverages real-world videos with humans removed and synthetic data containing virtual humans to pre-train a model. We then evaluate the transferability of the representation learned on this data to a diverse set of downstream action recognition benchmarks. Our approach outperforms previous baselines by up to 5%.
  arXiv  Detail & Related papers  (2023-11-10T18:38:14Z)
• Enhanced Human-Robot Collaboration using Constrained Probabilistic Human-Motion Prediction [5.501477817904299]
  We propose a novel human motion prediction framework that incorporates human joint constraints and scene constraints. It is tested on a human arm kinematic model and implemented on a human-robot collaborative setup with a UR5 robot arm.
  arXiv  Detail & Related papers  (2023-10-05T05:12:14Z)
• Model Predictive Control for Fluid Human-to-Robot Handovers [50.72520769938633]
  Planning motions that take human comfort into account is not a part of the human-robot handover process. We propose to generate smooth motions via an efficient model-predictive control framework. We conduct human-to-robot handover experiments on a diverse set of objects with several users.
  arXiv  Detail & Related papers  (2022-03-31T23:08:20Z)
• Hierarchical Graph-Convolutional Variational AutoEncoding for Generative Modelling of Human Motion [1.2599533416395767]
  Models of human motion commonly focus either on trajectory prediction or action classification but rarely both. Here we propose a novel architecture based on hierarchical variational autoencoders and deep graph convolutional neural networks for generating a holistic model of action over multiple time-scales. We show this Hierarchical Graph-conational Varivolutional Autoencoder (HG-VAE) to be capable of generating coherent actions, detecting out-of-distribution data, and imputing missing data by gradient ascent on the model's posterior.
  arXiv  Detail & Related papers  (2021-11-24T16:21:07Z)
• TRiPOD: Human Trajectory and Pose Dynamics Forecasting in the Wild [77.59069361196404]
  TRiPOD is a novel method for predicting body dynamics based on graph attentional networks. To incorporate a real-world challenge, we learn an indicator representing whether an estimated body joint is visible/invisible at each frame. Our evaluation shows that TRiPOD outperforms all prior work and state-of-the-art specifically designed for each of the trajectory and pose forecasting tasks.
  arXiv  Detail & Related papers  (2021-04-08T20:01:00Z)
• Residual Force Control for Agile Human Behavior Imitation and Extended Motion Synthesis [32.22704734791378]
  Reinforcement learning has shown great promise for realistic human behaviors by learning humanoid control policies from motion capture data. It is still very challenging to reproduce sophisticated human skills like ballet dance, or to stably imitate long-term human behaviors with complex transitions. We propose a novel approach, residual force control (RFC), that augments a humanoid control policy by adding external residual forces into the action space.
  arXiv  Detail & Related papers  (2020-06-12T17:56:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.