ACE-Ego-0: Unifying Egocentric Human and Robotic Data for VLA Pretraining

• URL: http://arxiv.org/abs/2606.17200v1
• Date: Mon, 15 Jun 2026 18:40:18 GMT
• Title: ACE-Ego-0: Unifying Egocentric Human and Robotic Data for VLA Pretraining
• Abstract summary: Vision-Language-Action (VLA) models benefit from large-scale and diverse embodied data, yet scaling robot trajectory collection is costly and labor-intensive.<n>Recent advances show that large-scale egocentric human videos provide complementary real-world supervision in pretraining.<n>We introduce ACE-EGO-0, a unified VLA pretraining framework jointly leveraging heterogeneous data sources.
• Score: 40.678451625177026
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision-Language-Action (VLA) models benefit from large-scale and diverse embodied data, yet scaling robot trajectory collection is costly and labor-intensive. Recent advances show that large-scale egocentric human videos provide complementary real-world supervision in pretraining. However, joint training on human and robot data remains challenging due to divergences in action spaces, embodiment structures, temporal dynamics, and supervision quality. We introduce ACE-EGO-0, a unified VLA pretraining framework jointly leveraging heterogeneous data sources. To extract large-scale pretraining supervision from egocentric human videos, we build a scalable egocentric video-to-action pipeline that converts raw human videos into robot-format pseudo-action trajectories. To make these labels comparable with robot demonstrations, ACE-EGO-0 uses a unified action representation based on camera-space actions, morphology conditioning, and time-aligned action chunking. To robustly leverage noisy pseudo-action supervision from egocentric human videos, we formulate a reliability-aware training objective with a human auxiliary loss that concentrates supervision on reliable signals. We instantiate ACE-EGO-0 on 4.53K hours of robot and simulation data, together with 1.48K hours of pseudo-action-labeled egocentric human data. Experiments show that incorporating large-scale human supervision under reliability-aware weighting consistently improves both unified joint pretraining and supervised fine-tuning. ACE-EGO-0 achieves state-of-the-art performance on RoboCasa GR1 TableTop and RoboTwin 2.0, while demonstrating strong transfer to real-world bimanual manipulation.

Related papers

• JoyAI-RA 0.5: Scaling Robot Manipulation Learning via Dual Action Alignment [0.0]
  We propose JoyAI-RA 0.5, a framework that couples physical world-dynamics with visual semantics and scales manipulation learning across such data.<n>Implicit action alignment infers latent actions from visual transitions, enabling action-free human, simulation, and robot data.<n>JoyAI-RA performs strongly on both seen tasks and unseen variations on a real-world AgiBot benchmark.
  arXiv  Detail & Related papers  (2026-08-06T07:14:55Z)
• Ego2Robot: Scalable Robot Data Synthesis from Egocentric Human Data [60.94418315274988]
  Ego2Robot supports both curated and in-the-wild videos, producing 18,561 hours of robot training data spanning 15 robot morphologies.<n>Joint pretraining on Ego2Robot-synthesized and robot data consistently improves out-of-distribution generalization across multiple types.
  arXiv  Detail & Related papers  (2026-08-03T17:52:26Z)
• ACE-Data-0: Human-Centric Ambient Capture as Embodied Data Engine [83.02804728003862]
  Ambient Capture Engine (ACE) transforms real home environments into spatially, temporally synchronized recording studios.<n>ACE records egocentric and multi-view exocentric video, full-body and articulated hand motion, object geometry and 6-DoF trajectories, audio, and signals as a unified multisensory stream.<n>Dataset spans atomic manipulation, long-horizon chains of household activities, and human-scene interaction, while preserving natural behavioral variation through goal-level rather than step-by-step instructions.
  arXiv  Detail & Related papers  (2026-07-30T17:59:50Z)
• Human-as-Humanoid: Enabling Zero-Shot Humanoid Learning from Ego-Exo Human Videos with Human-Aligned Embodiments [32.61025334275917]
  Human-as-Humanoid is a human-to-humanoid supervision framework for high-DoF humanoids.<n>It converts large-scale human demonstrations into executable observation--action supervision for the target humanoid.<n>Human-as-Humanoid yields a 4.8--7.2x raw demonstration-language gain over humanoid teleoperation in our data-collection analysis.
  arXiv  Detail & Related papers  (2026-06-30T17:44:16Z)
• ActiveMimic: Egocentric Video Pretraining with Active Perception [84.2999803878421]
  Egocentric human video offers a scalable alternative to robot data for pretraining.<n>We present ActiveMimic, a pretraining framework that recovers synchronized camera and wrist trajectories from a single body-worn RGB camera.<n>We show that ActiveMimic consistently surpasses baselines pretrained on human video and matches state-of-the-art models pretrained on robot data.
  arXiv  Detail & Related papers  (2026-06-04T14:01:01Z)
• Co-training with Ego-centric Video and Demonstration for Robot Navigation Task [0.0]
  We propose a framework that converts egocentric walking videos into datasets for mobile robot imitation learning.<n>The proposed method estimates camera motion from human videos and transforms it into action representations compatible with ground mobile robots.<n> Experiments on a fruit-search navigation task demonstrate that human egocentric videos provide an effective and scalable data source for mobile robot learning.
  arXiv  Detail & Related papers  (2026-06-01T09:12:22Z)
• $Ψ_0$: An Open Foundation Model Towards Universal Humanoid Loco-Manipulation [39.811210435945924]
  We introduce an open foundation model to address challenging humanoid loco-manipulation tasks.<n>Our research identifies a critical yet often overlooked data recipe.<n>We demonstrate that pre-training on high-quality egocentric human manipulation data followed by post-training on domain-specific real-world humanoid trajectories yields superior performance.
  arXiv  Detail & Related papers  (2026-03-12T17:59:51Z)
• EgoScale: Scaling Dexterous Manipulation with Diverse Egocentric Human Data [114.89243396877453]
  We present EgoScale, a human to dexterous manipulation transfer framework built on large scale egocentric human data.<n>We introduce a simple two stage transfer recipe: large scale human pretraining followed by lightweight aligned human robot mid training.<n>Our final policy improves average success rate by 54% over a no pretraining baseline using a 22 DoF dexterous robotic hand.
  arXiv  Detail & Related papers  (2026-02-18T18:59:05Z)
• EgoHumanoid: Unlocking In-the-Wild Loco-Manipulation with Robot-Free Egocentric Demonstration [67.13034606664333]
  EgoHumanoid is the first framework to co-train a vision-language-action policy using egocentric human demonstrations.<n>A portable system for scalable human data collection is developed.
  arXiv  Detail & Related papers  (2026-02-10T18:59:03Z)
• Scalable Vision-Language-Action Model Pretraining for Robotic Manipulation with Real-Life Human Activity Videos [42.86535655563404]
  We develop a fully-automated holistic human activity analysis approach for arbitrary human hand videos.<n>We process a large volume of egocentric videos and create a hand-VLA training dataset containing 1M episodes and 26M frames.<n>We design a dexterous hand VLA model architecture and pretrain the model on this dataset.
  arXiv  Detail & Related papers  (2025-10-24T15:39:31Z)
• MimicDreamer: Aligning Human and Robot Demonstrations for Scalable VLA Training [40.45924128424013]
  We propose MimicDreamer, a framework that turns low-cost human demonstrations into robot-usable supervision.<n>For visual alignment, we propose H2R Aligner, a video diffusion model that generates high-fidelity robot demonstration videos.<n>For viewpoint stabilization, EgoStabilizer is proposed, which canonicalizes egocentric videos via homography.<n>For action alignment, we map human hand trajectories to the robot frame and apply a constrained inverse kinematics solver.
  arXiv  Detail & Related papers  (2025-09-26T11:05:10Z)
• Being-H0: Vision-Language-Action Pretraining from Large-Scale Human Videos [66.62109400603394]
  We introduce Being-H0, a dexterous Vision-Language-Action model trained on large-scale human videos.<n>Our approach centers on physical instruction tuning, a novel training paradigm that combines large-scale VLA pretraining from human videos, physical space alignment for 3D reasoning, and post-training adaptation for robotic tasks.<n>We empirically show the excellence of Being-H0 in hand motion generation and instruction following, and it also scales well with model and data sizes.
  arXiv  Detail & Related papers  (2025-07-21T13:19:09Z)
• EgoVLA: Learning Vision-Language-Action Models from Egocentric Human Videos [49.820119587446655]
  In this paper, we explore training Vision-Language-Action (VLA) models using egocentric human videos.<n>With a VLA trained on human video that predicts human wrist and hand actions, we can perform Inverse Kinematics and convert the human actions to robot actions.<n>We propose a simulation benchmark called Ego Humanoid Manipulation Benchmark, where we design diverse bimanual manipulation tasks with demonstrations.
  arXiv  Detail & Related papers  (2025-07-16T17:27:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.