EgoEngine: From Egocentric Human Videos to High-Fidelity Dexterous Robot Demonstrations

• URL: http://arxiv.org/abs/2606.12604v1
• Date: Wed, 10 Jun 2026 19:01:40 GMT
• Title: EgoEngine: From Egocentric Human Videos to High-Fidelity Dexterous Robot Demonstrations
• Abstract summary: EgoEngine is a framework for transforming egocentric human manipulation videos into high-fidelity robot data.<n>We show that EgoEngine enables scalable conversion of human videos into robot data and, to our knowledge, demonstrates the first zero-shot visuomotor dexterous policy learning from egocentric human videos.
• Score: 12.604831042368078
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Dexterous manipulation is limited by the cost of collecting large-scale robot demonstrations. Egocentric human videos offer a scalable source of diverse manipulation behaviors, but directly using them for robot learning requires bridging two gaps: the visual gap between human and robot observations, and the action gap between human motion and robot-executable action. We propose EgoEngine, a scalable framework for transforming egocentric human manipulation videos into high-fidelity robot data. Given an egocentric RGB video, EgoEngine produces: (i) a high-fidelity robot observation video replacing human with robot while preserving scene context and temporal alignment, and (ii) a task-aligned, executable robot action trajectory under feasibility constraints. Experiments in simulation and on real robots show that EgoEngine enables scalable conversion of human videos into robot data and, to our knowledge, demonstrates the first zero-shot visuomotor dexterous policy learning from egocentric human videos without real-robot demonstrations. Project website: https://egoengine.github.io.

Related papers

• Ego2Robot: Scalable Robot Data Synthesis from Egocentric Human Data [60.94418315274988]
  Ego2Robot supports both curated and in-the-wild videos, producing 18,561 hours of robot training data spanning 15 robot morphologies.<n>Joint pretraining on Ego2Robot-synthesized and robot data consistently improves out-of-distribution generalization across multiple types.
  arXiv  Detail & Related papers  (2026-08-03T17:52:26Z)
• HumanEgo: Zero-Shot Robot Learning from Minutes of Human Egocentric Videos [58.9564236347451]
  HumanEgo is a framework that bridges the embodiment gap between human and robot.<n>It lifts each human demonstration to an entity-level representation of hand-object interaction.<n>HumanEgo is robot-data-free, hardware-agnostic, data-efficient, and zero-shot human-to-robot transferable.
  arXiv  Detail & Related papers  (2026-05-24T08:26:41Z)
• H2R-Grounder: A Paired-Data-Free Paradigm for Translating Human Interaction Videos into Physically Grounded Robot Videos [58.006918399913665]
  We propose a video-to-video translation framework that converts ordinary human-object interaction videos into motion-consistent robot manipulation videos.<n>Our approach does not require any paired human-robot videos for training only a set of unpaired robot videos, making the system easy to scale.<n>At test time, we apply the same process to human videos (inpainting the person and overlaying human pose cues) and generate high-quality robot videos that mimic the human's actions.
  arXiv  Detail & Related papers  (2025-12-10T07:59:45Z)
• EgoVLA: Learning Vision-Language-Action Models from Egocentric Human Videos [49.820119587446655]
  In this paper, we explore training Vision-Language-Action (VLA) models using egocentric human videos.<n>With a VLA trained on human video that predicts human wrist and hand actions, we can perform Inverse Kinematics and convert the human actions to robot actions.<n>We propose a simulation benchmark called Ego Humanoid Manipulation Benchmark, where we design diverse bimanual manipulation tasks with demonstrations.
  arXiv  Detail & Related papers  (2025-07-16T17:27:44Z)
• EgoZero: Robot Learning from Smart Glasses [54.6168258133554]
  EgoZero learns robust manipulation policies from human demonstrations captured with Project Aria smart glasses.<n>We deploy EgoZero policies on a Franka Panda robot and demonstrate zero-shot transfer with 70% success rate over 7 manipulation tasks.<n>Our results suggest that in-the-wild human data can serve as a scalable foundation for real-world robot learning.
  arXiv  Detail & Related papers  (2025-05-26T17:59:17Z)
• Generalizable Humanoid Manipulation with 3D Diffusion Policies [66.78220965526732]
  We build a real-world robotic system to address the problem of autonomous manipulation by humanoid robots.<n>Our system is mainly an integration of 1) a whole-upper-body robotic teleoperation system to acquire human-like robot data, and 2) a 25-DoF humanoid robot platform with a height-adjustable cart and a 3D LiDAR sensor.<n>We show that using only data collected in one scene and with only onboard computing, a full-sized humanoid robot can autonomously perform skills in diverse real-world scenarios.
  arXiv  Detail & Related papers  (2024-10-14T17:59:00Z)
• Vid2Robot: End-to-end Video-conditioned Policy Learning with Cross-Attention Transformers [36.497624484863785]
  We introduce Vid2Robot, an end-to-end video-conditioned policy that takes human videos demonstrating manipulation tasks as input and produces robot actions. Our model is trained with a large dataset of prompt video-robot trajectory pairs to learn unified representations of human and robot actions from videos. We evaluate Vid2Robot on real-world robots and observe over 20% improvement over BC-Z when using human prompt videos.
  arXiv  Detail & Related papers  (2024-03-19T17:47:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.