SUGAR: A Scalable Human-Video-Driven Generalizable Humanoid Loco-Manipulation Learning Framework

• URL: http://arxiv.org/abs/2605.20373v1
• Date: Tue, 19 May 2026 18:24:05 GMT
• Title: SUGAR: A Scalable Human-Video-Driven Generalizable Humanoid Loco-Manipulation Learning Framework
• Abstract summary: Building humanoid robots capable of generalizable whole-body loco-manipulation in the real world remains a fundamental challenge.<n>We present SUGAR, a scalable data-driven framework that converts diverse human videos into deployable humanoid loco-manipulation skills.<n>We evaluate SUGAR on six representative loco-manipulation tasks in simulation and real-world humanoid hardware.
• Score: 27.050974194855964
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Building humanoid robots capable of generalizable whole-body loco-manipulation in the real world remains a fundamental challenge. Existing methods either rely on laborious task-specific reward engineering, rigidly replay reference motions that fail to generalize, or depend on costly teleoperation that limits scalability. While human videos capture diverse human behaviors, motion priors inferred from them are inherently imperfect, suffering from occlusion, contact artifacts, and retargeting errors that render them unsuitable for direct policy learning. To address this, we present SUGAR, a scalable data-driven framework that converts diverse human videos into deployable humanoid loco-manipulation skills, without any task-specific reward engineering or reference-motion conditioning at inference. SUGAR proceeds in three stages. First, a fully automated pipeline extracts kinematic interaction priors including human-object motion trajectories and contact labels from unstructured human videos. Second, a privileged physics-based refiner uses a unified mimic reward and progressive state pool to transform imperfect priors into physically feasible, high-fidelity skills. Third, refined skills are distilled into a hierarchical autonomous policy consisting of a command generator and a command tracker. We evaluate SUGAR on six representative loco-manipulation tasks in simulation and real-world humanoid hardware. Our method substantially outperforms reference-tracking baselines, and performance scales clearly with the amount of human video data. It also achieves zero-shot real-world transfer with reliable closed-loop execution, autonomous failure recovery, and stable long-horizon performance under external perturbations. Project Page: https://tianshuwu.github.io/sugar-humanoid/

Related papers

• JoyAI-RA 0.5: Scaling Robot Manipulation Learning via Dual Action Alignment [0.0]
  We propose JoyAI-RA 0.5, a framework that couples physical world-dynamics with visual semantics and scales manipulation learning across such data.<n>Implicit action alignment infers latent actions from visual transitions, enabling action-free human, simulation, and robot data.<n>JoyAI-RA performs strongly on both seen tasks and unseen variations on a real-world AgiBot benchmark.
  arXiv  Detail & Related papers  (2026-08-06T07:14:55Z)
• RoboReact: Agentic Skill Distillation from Generated Egocentric Videos for Generalizable Whole-Body Manipulation [18.992624318431417]
  We present RoboReact, a framework that automatically synthesizes whole-body humanoid manipulation skills from a single egocentric RGB-D observation.<n>RoboReact generates human manipulation videos, extracts geometry interactions through depth-aware 3D reconstruction, and retargets them to high-DoF humanoid platforms while preserving hand-object interaction geometry.<n>Experiments on real humanoid robots demonstrate that RoboReact generalizes across diverse object configurations and robustly recovers from execution disturbances without requiring teleoperation or human demonstrations.
  arXiv  Detail & Related papers  (2026-08-04T09:37:06Z)
• Human-as-Humanoid: Enabling Zero-Shot Humanoid Learning from Ego-Exo Human Videos with Human-Aligned Embodiments [32.61025334275917]
  Human-as-Humanoid is a human-to-humanoid supervision framework for high-DoF humanoids.<n>It converts large-scale human demonstrations into executable observation--action supervision for the target humanoid.<n>Human-as-Humanoid yields a 4.8--7.2x raw demonstration-language gain over humanoid teleoperation in our data-collection analysis.
  arXiv  Detail & Related papers  (2026-06-30T17:44:16Z)
• $Ψ_0$: An Open Foundation Model Towards Universal Humanoid Loco-Manipulation [39.811210435945924]
  We introduce an open foundation model to address challenging humanoid loco-manipulation tasks.<n>Our research identifies a critical yet often overlooked data recipe.<n>We demonstrate that pre-training on high-quality egocentric human manipulation data followed by post-training on domain-specific real-world humanoid trajectories yields superior performance.
  arXiv  Detail & Related papers  (2026-03-12T17:59:51Z)
• ZeroWBC: Learning Natural Visuomotor Humanoid Control Directly from Human Egocentric Video [52.78703020909145]
  We introduce ZeroWBC, a novel framework that learns a natural humanoid visuomotor control policy directly from human egocentric videos.<n>Our approach first fine-tunes a Vision-Language Model (VLM) to predict future whole-body human motions based on text instructions and egocentric visual context.<n>Experiments on the Unitree G1 humanoid robot demonstrate that our method outperforms baseline approaches in motion naturalness and versatility.
  arXiv  Detail & Related papers  (2026-03-10T04:19:43Z)
• EgoHumanoid: Unlocking In-the-Wild Loco-Manipulation with Robot-Free Egocentric Demonstration [67.13034606664333]
  EgoHumanoid is the first framework to co-train a vision-language-action policy using egocentric human demonstrations.<n>A portable system for scalable human data collection is developed.
  arXiv  Detail & Related papers  (2026-02-10T18:59:03Z)
• DexImit: Learning Bimanual Dexterous Manipulation from Monocular Human Videos [56.64773686434068]
  We propose DexImit, an automated framework that converts human manipulation videos into physically plausible robot data.<n>DexImit can generate large-scale robot data based on human videos, either from the Internet or video generation models.<n>It is capable of handling diverse manipulation tasks, including tool use, long-horizon tasks, and fine-grained manipulations.
  arXiv  Detail & Related papers  (2026-02-10T18:59:02Z)
• HumanX: Toward Agile and Generalizable Humanoid Interaction Skills from Human Videos [74.43500240121476]
  We present HumanX, a full-stack framework that compiles human video into generalizable, real-world interaction skills for humanoids.<n>HumanX achieves over 8 times higher generalization success than prior methods.
  arXiv  Detail & Related papers  (2026-02-02T18:53:01Z)
• From Generated Human Videos to Physically Plausible Robot Trajectories [103.28274349461607]
  Video generation models are rapidly improving in their ability to synthesize human actions in novel contexts.<n>To realize this potential, how can a humanoid execute the human actions from generated videos in a zero-shot manner?<n>This challenge arises because generated videos are often noisy and exhibit morphological distortions that make direct imitation difficult compared to real video.<n>We propose GenMimic, a physics-aware reinforcement learning policy conditioned on 3D keypoints, and trained with symmetry regularization and keypoint-weighted tracking rewards.
  arXiv  Detail & Related papers  (2025-12-04T18:56:03Z)
• ResMimic: From General Motion Tracking to Humanoid Whole-body Loco-Manipulation via Residual Learning [59.64325421657381]
  Humanoid whole-body loco-manipulation promises transformative capabilities for daily service and warehouse tasks.<n>We introduce ResMimic, a two-stage residual learning framework for precise and expressive humanoid control from human motion data.<n>Results show substantial gains in task success, training efficiency, and robustness over strong baselines.
  arXiv  Detail & Related papers  (2025-10-06T17:47:02Z)
• Learning a Universal Human Prior for Dexterous Manipulation from Human Preference [35.54663426598218]
  We propose a framework that learns a universal human prior using direct human preference feedback over videos. A task-agnostic reward model is trained through iteratively generating diverse polices and collecting human preference over the trajectories. Our method empirically demonstrates more human-like behaviors on robot hands in diverse tasks including even unseen tasks.
  arXiv  Detail & Related papers  (2023-04-10T14:17:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.