HumanNet: Scaling Human-centric Video Learning to One Million Hours

• URL: http://arxiv.org/abs/2605.06747v1
• Date: Thu, 07 May 2026 15:21:58 GMT
• Title: HumanNet: Scaling Human-centric Video Learning to One Million Hours
• Abstract summary: We present HumanNet, a one-million-hour human-centric video corpus that captures how humans interact with the physical world at scale.<n>HumanNet spans both first-person and third-person perspectives and covers fine-grained activities, human-object interactions, tool use, and long-horizon behaviors.<n>Beyond raw video, the dataset provides interaction-centric annotations, including captions, motion descriptions, and hand and body-related signals.
• Score: 18.004001748422407
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Progress in embodied intelligence increasingly depends on scalable data infrastructure. While vision and language have scaled with internet corpora, learning physical interaction remains constrained by the lack of large, diverse, and richly annotated human activity data. We present HumanNet, a one-million-hour human-centric video corpus that captures how humans interact with the physical world at scale. HumanNet spans both first-person and third-person perspectives and covers fine-grained activities, human-object interactions, tool use, and long-horizon behaviors across diverse real-world environments. Beyond raw video, the dataset provides interaction-centric annotations, including captions, motion descriptions, and hand and body-related signals, enabling motion-aware and interaction-aware learning. Beyond scale, HumanNet introduces a systematic data curation paradigm for embodied learning, where human-centric filtering, temporal structuring, viewpoint diversity, and annotation enrichment are treated as first-class design principles. This design transforms unstructured internet video into a scalable substrate for representation learning, activity understanding, motion generation, and human-to-robot transfer. We conduct a first-step validation on the value of this design through controlled vision-language-action ablation: under a fixed set of validation data, continued training from the Qwen VLM model with 1000 hours of egocentric video drawn from HumanNet surpasses the continued training with 100 hours of real-robot data from Magic Cobot, indicating that egocentric human video could be a scalable and cost-effective substitute for robot data. By building this project, we aim to explore the opportunity to scale embodied foundation models using human-centric videos, rather than relying solely on robot-specific data.

Related papers

• Ego2Robot: Scalable Robot Data Synthesis from Egocentric Human Data [60.94418315274988]
  Ego2Robot supports both curated and in-the-wild videos, producing 18,561 hours of robot training data spanning 15 robot morphologies.<n>Joint pretraining on Ego2Robot-synthesized and robot data consistently improves out-of-distribution generalization across multiple types.
  arXiv  Detail & Related papers  (2026-08-03T17:52:26Z)
• EgoVerse: An Egocentric Human Dataset for Robot Learning from Around the World [63.2070298441355]
  EgoVerse is a collaborative platform for human data-driven robot learning.<n>We release 1,362 hours (80k episodes) of human demonstrations spanning 1,965 tasks, 240 scenes, and 2,087 unique demonstrators.<n>We conduct a large-scale study of human-to-robot transfer with experiments replicated across multiple labs, tasks, and robot embodiments.
  arXiv  Detail & Related papers  (2026-04-08T21:27:06Z)
• $Ψ_0$: An Open Foundation Model Towards Universal Humanoid Loco-Manipulation [39.811210435945924]
  We introduce an open foundation model to address challenging humanoid loco-manipulation tasks.<n>Our research identifies a critical yet often overlooked data recipe.<n>We demonstrate that pre-training on high-quality egocentric human manipulation data followed by post-training on domain-specific real-world humanoid trajectories yields superior performance.
  arXiv  Detail & Related papers  (2026-03-12T17:59:51Z)
• EgoHumanoid: Unlocking In-the-Wild Loco-Manipulation with Robot-Free Egocentric Demonstration [67.13034606664333]
  EgoHumanoid is the first framework to co-train a vision-language-action policy using egocentric human demonstrations.<n>A portable system for scalable human data collection is developed.
  arXiv  Detail & Related papers  (2026-02-10T18:59:03Z)
• HumanX: Toward Agile and Generalizable Humanoid Interaction Skills from Human Videos [74.43500240121476]
  We present HumanX, a full-stack framework that compiles human video into generalizable, real-world interaction skills for humanoids.<n>HumanX achieves over 8 times higher generalization success than prior methods.
  arXiv  Detail & Related papers  (2026-02-02T18:53:01Z)
• Emergence of Human to Robot Transfer in Vision-Language-Action Models [88.76648919814771]
  Vision-language-action (VLA) models can enable broad open world generalization, but require large and diverse datasets.<n>We show that human-to-robot transfer emerges once the VLA is pre-trained on sufficient scenes, tasks, and embodiments.
  arXiv  Detail & Related papers  (2025-12-27T00:13:11Z)
• Scalable Vision-Language-Action Model Pretraining for Robotic Manipulation with Real-Life Human Activity Videos [42.86535655563404]
  We develop a fully-automated holistic human activity analysis approach for arbitrary human hand videos.<n>We process a large volume of egocentric videos and create a hand-VLA training dataset containing 1M episodes and 26M frames.<n>We design a dexterous hand VLA model architecture and pretrain the model on this dataset.
  arXiv  Detail & Related papers  (2025-10-24T15:39:31Z)
• Humanoid Everyday: A Comprehensive Robotic Dataset for Open-World Humanoid Manipulation [16.701354625940308]
  Humanoid Everyday is a large-scale and diverse humanoid manipulation dataset.<n>It aggregates high-quality multimodal sensory data, including RGB, depth, LiDAR, and tactile inputs, together with natural language annotations.<n>We conduct an analysis of representative policy learning methods on our dataset, providing insights into their strengths and limitations.
  arXiv  Detail & Related papers  (2025-10-09T20:43:27Z)
• Being-H0: Vision-Language-Action Pretraining from Large-Scale Human Videos [66.62109400603394]
  We introduce Being-H0, a dexterous Vision-Language-Action model trained on large-scale human videos.<n>Our approach centers on physical instruction tuning, a novel training paradigm that combines large-scale VLA pretraining from human videos, physical space alignment for 3D reasoning, and post-training adaptation for robotic tasks.<n>We empirically show the excellence of Being-H0 in hand motion generation and instruction following, and it also scales well with model and data sizes.
  arXiv  Detail & Related papers  (2025-07-21T13:19:09Z)
• EgoMimic: Scaling Imitation Learning via Egocentric Video [22.902881956495765]
  We present EgoMimic, a full-stack framework which scales manipulation via human embodiment data. EgoMimic achieves this through: (1) a system to capture human embodiment data using the ergonomic Project Aria glasses, (2) a low-cost bimanual manipulator that minimizes the kinematic gap to human data, and (4) an imitation learning architecture that co-trains on human and robot data.
  arXiv  Detail & Related papers  (2024-10-31T17:59:55Z)
• Learning Predictive Models From Observation and Interaction [137.77887825854768]
  Learning predictive models from interaction with the world allows an agent, such as a robot, to learn about how the world works. However, learning a model that captures the dynamics of complex skills represents a major challenge. We propose a method to augment the training set with observational data of other agents, such as humans.
  arXiv  Detail & Related papers  (2019-12-30T01:10:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.