HALOMI: Learning Humanoid Loco-Manipulation with Active Perception from Human Demonstrations

• URL: http://arxiv.org/abs/2606.18772v1
• Date: Wed, 17 Jun 2026 07:33:37 GMT
• Title: HALOMI: Learning Humanoid Loco-Manipulation with Active Perception from Human Demonstrations
• Abstract summary: We present HALOMI, a framework for learning humanoid loco-manipulation with active perception from human demonstrations.<n>We propose a manifold-constrained controller that plans in a learned latent behavior manifold to enable precise and robust head-hand tracking in the world frame.<n>We validate HALOMI on a Unitree G1 humanoid robot with an actuated neck across five real-world tasks involving navigation, grasping, bimanual manipulation, whole-body coordination, and dynamic behaviors.
• Score: 10.378782873235453
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Human demonstrations, which can be collected at scale and naturally capture active hand-eye coordination, are a promising data source for learning humanoid loco-manipulation. However, directly transferring human demonstrations to humanoids requires a precise world-frame tracking controller, which is often brittle under Out-of-Distribution(OOD) targets, while human-to-humanoid gaps persist in both egocentric observation and action execution. To address these challenges, we present HALOMI, a scalable framework for learning humanoid loco-manipulation with active perception from human demonstrations. HALOMI extends Universal Manipulation Interface (UMI) with egocentric sensing to collect ego-view and wrist-view observations along with head-hand trajectories at scale. We further propose a manifold-constrained controller that plans in a learned latent behavior manifold to enable precise and robust head-hand tracking in the world frame. To bridge the human-to-humanoid gap, we perform ego-view alignment and introduce a controller-aware reference trajectory adaptation to reduce mismatch in both observation and action execution. We validate HALOMI on a Unitree G1 humanoid robot with an actuated neck across five real-world tasks involving navigation, grasping, bimanual manipulation, whole-body coordination, and dynamic behaviors. Across the three quantitatively evaluated tasks, HALOMI achieves an average success rate of 85\%, while additional qualitative demonstrations show its ability to support dynamic tossing and deep-squat grasping.

Related papers

• DreamMimic: Learning Visuomotor Whole-Body Loco-Manipulation via World Model [5.082137967180834]
  Vision-based whole-body loco-manipulation on humanoid robots is challenging due to partial observability, contact-rich dynamics, and the difficulty of learning long-horizon behaviors from high-dimensional visual inputs.<n>We present a framework that distills privileged teacher policies into vision-based humanoid controllers via world-model-assisted distillation.
  arXiv  Detail & Related papers  (2026-08-23T08:08:39Z)
• Human-as-Humanoid: Enabling Zero-Shot Humanoid Learning from Ego-Exo Human Videos with Human-Aligned Embodiments [32.61025334275917]
  Human-as-Humanoid is a human-to-humanoid supervision framework for high-DoF humanoids.<n>It converts large-scale human demonstrations into executable observation--action supervision for the target humanoid.<n>Human-as-Humanoid yields a 4.8--7.2x raw demonstration-language gain over humanoid teleoperation in our data-collection analysis.
  arXiv  Detail & Related papers  (2026-06-30T17:44:16Z)
• HumanoidUMI: Bridging Robot-Free Demonstrations and Humanoid Whole-Body Manipulation [6.733928872257592]
  We propose HumanoidUMI, a portable and robot-free framework for humanoid whole-body data collection.<n>We use lightweight VR devices and UMI-inspired grippers to collect sparse human keypoint trajectories, wrist-view observations, and gripper actions.<n>Experiments in five real-world scenarios demonstrate the effectiveness of the proposed framework.
  arXiv  Detail & Related papers  (2026-06-25T16:23:56Z)
• HumanoidMimicGen: Data Generation for Loco-Manipulation via Whole-Body Planning [52.022681285103126]
  We present HumanoidMimicGen, a method for generating humanoid legged loco-manipulation data.<n>Our method adapts contact-rich whole-body skills from a handful of source demonstrations to new states.<n>We show that whole-body visuomotor policies co-trained with data generated by HumanoidMimicGen outperform those trained only on real-world data by 20%.
  arXiv  Detail & Related papers  (2026-05-26T21:57:11Z)
• HEX: Humanoid-Aligned Experts for Cross-Embodiment Whole-Body Manipulation [74.34984994596813]
  HEX is a state-centric framework for coordinated manipulation on humanoid robots.<n>It incorporates a humanoid-aligned universal state representation for scalable learning across heterogeneous embodiments.<n>It achieves state-of-the-art performance in task success rate and generalization.
  arXiv  Detail & Related papers  (2026-04-09T09:01:43Z)
• ZeroWBC: Learning Natural Visuomotor Humanoid Control Directly from Human Egocentric Video [52.78703020909145]
  We introduce ZeroWBC, a novel framework that learns a natural humanoid visuomotor control policy directly from human egocentric videos.<n>Our approach first fine-tunes a Vision-Language Model (VLM) to predict future whole-body human motions based on text instructions and egocentric visual context.<n>Experiments on the Unitree G1 humanoid robot demonstrate that our method outperforms baseline approaches in motion naturalness and versatility.
  arXiv  Detail & Related papers  (2026-03-10T04:19:43Z)
• ULTRA: Unified Multimodal Control for Autonomous Humanoid Whole-Body Loco-Manipulation [55.467742403416175]
  We introduce a physics-driven neural algorithm that translates large-scale motion capture to humanoid embodiments.<n>We learn a unified multimodal controller that supports both dense references and sparse task specifications.<n>Results show that ULTRA generalizes to autonomous, goal-conditioned whole-body loco-manipulation from egocentric perception.
  arXiv  Detail & Related papers  (2026-03-03T18:59:29Z)
• EgoHumanoid: Unlocking In-the-Wild Loco-Manipulation with Robot-Free Egocentric Demonstration [67.13034606664333]
  EgoHumanoid is the first framework to co-train a vision-language-action policy using egocentric human demonstrations.<n>A portable system for scalable human data collection is developed.
  arXiv  Detail & Related papers  (2026-02-10T18:59:03Z)
• METIS: Multi-Source Egocentric Training for Integrated Dexterous Vision-Language-Action Model [36.82365894983052]
  A major bottleneck lies in the scarcity of large-scale, action-annotated data for dexterous skills.<n>We propose METIS, a vision-language-action model for dexterous manipulation pretrained on egocentric datasets.<n>Our method demonstrates exceptional dexterous manipulation capabilities, achieving highest average success rate in six real-world tasks.
  arXiv  Detail & Related papers  (2025-11-21T16:32:36Z)
• DemoHLM: From One Demonstration to Generalizable Humanoid Loco-Manipulation [29.519071338337685]
  We present DemoHLM, a framework for humanoid loco-manipulation on a real humanoid robot from a single demonstration in simulation.<n>whole-body controller maps whole-body motion commands to joint torques and provides omnidirectional mobility for the humanoid robot.<n> Experiments show a positive correlation between the amount of synthetic data and policy performance.
  arXiv  Detail & Related papers  (2025-10-13T10:49:40Z)
• ResMimic: From General Motion Tracking to Humanoid Whole-body Loco-Manipulation via Residual Learning [59.64325421657381]
  Humanoid whole-body loco-manipulation promises transformative capabilities for daily service and warehouse tasks.<n>We introduce ResMimic, a two-stage residual learning framework for precise and expressive humanoid control from human motion data.<n>Results show substantial gains in task success, training efficiency, and robustness over strong baselines.
  arXiv  Detail & Related papers  (2025-10-06T17:47:02Z)
• Learning Multi-Modal Whole-Body Control for Real-World Humanoid Robots [13.229028132036321]
  Masked Humanoid Controller (MHC) supports standing, walking, and mimicry of whole and partial-body motions.<n>MHC imitates partially masked motions from a library of behaviors spanning standing, walking, optimized reference trajectories, re-targeted video clips, and human motion capture data.<n>We demonstrate sim-to-real transfer on the real-world Digit V3 humanoid robot.
  arXiv  Detail & Related papers  (2024-07-30T09:10:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.