EgoSteer: A Full-Stack System Towards Steerable Dexterous Manipulation from Egocentric Videos

• URL: http://arxiv.org/abs/2607.09701v1
• Date: Sun, 21 Jun 2026 16:16:02 GMT
• Title: EgoSteer: A Full-Stack System Towards Steerable Dexterous Manipulation from Egocentric Videos
• Abstract summary: We present a full-stack system that scales dexterous VLA pre-training from egocentric human videos.<n>It integrates EgoSmith, a data pipeline that curates in-the-wild egocentric videos into 9.6K hours of high-quality pre-training data.<n>EgoSteer robustly executes free-form instructions across 40+ diverse tasks, demonstrating failure recovery, dexterity, and generalization.
• Score: 27.122239961936756
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Steerability is a defining capability of generalist robot policies, yet remains largely absent in dexterous-hand systems for lack of large-scale, language-aligned, and action-accurate demonstration data. To address this bottleneck, we present a full-stack system that scales dexterous VLA pre-training from egocentric human videos and enables data-efficient real-robot post-training. It integrates EgoSmith, a data pipeline that curates in-the-wild egocentric videos into 9.6K hours of high-quality pre-training data with 9x higher throughput and better accuracy than prior SOTA; a unified robot stack for teleoperation and human-in-the-loop correction; and EgoSteer, a world-model-enhanced VLA trained on optimized infrastructure. Human-data pre-training equips EgoSteer with language-guided manipulation priors, which are grounded through robot post-training and improved by DAgger refinement. Empirically, EgoSteer robustly executes free-form instructions across 40+ diverse tasks, demonstrating failure recovery, dexterity, and generalization. The pre-trained model also few-shot adapts to complex long-horizon tasks, including box folding, on two embodiments with 75+% success. We open-source the system, data, and model at https://egosteer.github.io/.

Related papers

• Ego2Robot: Scalable Robot Data Synthesis from Egocentric Human Data [60.94418315274988]
  Ego2Robot supports both curated and in-the-wild videos, producing 18,561 hours of robot training data spanning 15 robot morphologies.<n>Joint pretraining on Ego2Robot-synthesized and robot data consistently improves out-of-distribution generalization across multiple types.
  arXiv  Detail & Related papers  (2026-08-03T17:52:26Z)
• Robot Self-Improvement via Human-Video Dynamics Models [52.14862052988773]
  We show that human videos can be used to learn embodiment-agnostic action, dynamics, and value representations that transfer across robot embodiments.<n>We introduce Dynamics-Guided Action Correction (DGAC), a training-free approach that leverages these adapted models to repair failed states.<n>Results show that human priors and robot failures can be combined to enable scalable autonomous policy improvement.
  arXiv  Detail & Related papers  (2026-06-19T13:17:27Z)
• HumanScale: Egocentric Human Video Can Outperform Real-Robot Data for Embodied Pretraining [93.5192770515694]
  Embodied foundation models are expected to benefit from data scaling like large language models, but face a much tighter data bottleneck.<n>Teleoperated real-robot trajectories remain the dominant pretraining source due to their precise action supervision and embodiment alignment.<n>Embodied foundation models pretrained on egocentric data achieve a 24% lower validation loss on real-robot action prediction.
  arXiv  Detail & Related papers  (2026-06-18T17:37:34Z)
• Contrastive Action-Image Pre-training for Visuomotor Control [91.5630572541332]
  We introduce CAIP, a vision encoder that treats human hand poses from large-scale egocentric video as a proxy for end-effector actions.<n>Leveraging 32,041 hours of egocentric human video and only 88 hours of robotic manipulation data, CAIP outperforms state-of-the-art vision encoders.
  arXiv  Detail & Related papers  (2026-06-15T20:00:20Z)
• ACE-Ego-0: Unifying Egocentric Human and Robotic Data for VLA Pretraining [40.678451625177026]
  Vision-Language-Action (VLA) models benefit from large-scale and diverse embodied data, yet scaling robot trajectory collection is costly and labor-intensive.<n>Recent advances show that large-scale egocentric human videos provide complementary real-world supervision in pretraining.<n>We introduce ACE-EGO-0, a unified VLA pretraining framework jointly leveraging heterogeneous data sources.
  arXiv  Detail & Related papers  (2026-06-15T18:40:18Z)
• EgoScale: Scaling Dexterous Manipulation with Diverse Egocentric Human Data [114.89243396877453]
  We present EgoScale, a human to dexterous manipulation transfer framework built on large scale egocentric human data.<n>We introduce a simple two stage transfer recipe: large scale human pretraining followed by lightweight aligned human robot mid training.<n>Our final policy improves average success rate by 54% over a no pretraining baseline using a 22 DoF dexterous robotic hand.
  arXiv  Detail & Related papers  (2026-02-18T18:59:05Z)
• PhysBrain: Human Egocentric Data as a Bridge from Vision Language Models to Physical Intelligence [19.558594034613996]
  Egocentric2Embodiment Translation Pipeline transforms raw human egocentric videos into multi-level, schema-driven embodiment supervision.<n>An egocentric-aware embodied brain, PhysBrain, is obtained by training on the E2E-3M dataset.
  arXiv  Detail & Related papers  (2025-12-18T17:27:03Z)
• Developing Vision-Language-Action Model from Egocentric Videos [14.1517430035289]
  Egocentric videos capture how humans manipulate objects and tools, providing diverse motion cues for learning object manipulation.<n>Prior studies that leverage such videos for training robot policies typically rely on auxiliary annotations, such as detailed hand-pose recordings.<n>In this work, we address this challenge by leveraging EgoScaler, a framework that extracts 6DoF object manipulation trajectories from egocentric videos.
  arXiv  Detail & Related papers  (2025-09-26T07:09:33Z)
• Being-H0: Vision-Language-Action Pretraining from Large-Scale Human Videos [66.62109400603394]
  We introduce Being-H0, a dexterous Vision-Language-Action model trained on large-scale human videos.<n>Our approach centers on physical instruction tuning, a novel training paradigm that combines large-scale VLA pretraining from human videos, physical space alignment for 3D reasoning, and post-training adaptation for robotic tasks.<n>We empirically show the excellence of Being-H0 in hand motion generation and instruction following, and it also scales well with model and data sizes.
  arXiv  Detail & Related papers  (2025-07-21T13:19:09Z)
• Robotic Offline RL from Internet Videos via Value-Function Pre-Training [67.44673316943475]
  We develop a system for leveraging large-scale human video datasets in robotic offline RL. We show that value learning on video datasets learns representations more conducive to downstream robotic offline RL than other approaches.
  arXiv  Detail & Related papers  (2023-09-22T17:59:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.