EgoExo-WM: Unlocking Exo Video for Ego World Models

• URL: http://arxiv.org/abs/2605.15477v1
• Date: Thu, 14 May 2026 23:35:54 GMT
• Title: EgoExo-WM: Unlocking Exo Video for Ego World Models
• Abstract summary: Egocentric world models present a promising direction for enabling agents to predict and plan.<n>Exocentric video is abundant and reveals body poses well, but lacks direct alignment with an agent's action space.<n>We propose a method to bridge this gap by extracting structured body pose from exocentric video as a representation of action.
• Score: 59.337519140691775
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Egocentric world models present a promising direction for enabling agents to predict and plan, but their performance is constrained by the limited availability of egocentric training data and its inherent partial observability of humans' physical actions. In contrast, exocentric video is abundant and reveals body poses well, but lacks direct alignment with an agent's action space -- and is not egocentric. We propose a method to bridge this gap by extracting structured body pose from exocentric video as a representation of action and transforming the exocentric video to egocentric video, informed by a human kinematics prior. This process unlocks the integration of in-the-wild exocentric data for egocentric world model training. We show that training whole-body action-conditioned egocentric world models with our converted data significantly improves both prediction quality and downstream planning performance, where we infer the sequence of body poses needed to achieve a visual goal state. Our approach paves the way to enlist arbitrary in-the-wild videos for building powerful egocentric world models, furthering applications in robot planning and augmented-reality guidance.

Related papers

• EgoX: Egocentric Video Generation from a Single Exocentric Video [46.41583107241048]
  We present EgoX, a novel framework for generating egocentric videos from a single excentrico input.<n>Our approach achieves coherent and realistic egocentric video generation while demonstrating strong scalability and robustness across unseen in-the-wild videos.
  arXiv  Detail & Related papers  (2025-12-09T05:53:39Z)
• Whole-Body Conditioned Egocentric Video Prediction [98.94980209293776]
  We train models to Predict Ego-centric Video from human Actions (PEVA)<n>By conditioning on kinematic pose trajectories, structured by the joint hierarchy of the body, our model learns to simulate how physical human actions shape the environment from a first-person point of view.<n>Our work represents an initial attempt to tackle the challenges of modeling complex real-world environments and embodied agent behaviors with video prediction from the perspective of a human.
  arXiv  Detail & Related papers  (2025-06-26T17:59:59Z)
• PlayerOne: Egocentric World Simulator [73.88786358213694]
  PlayerOne is the first egocentric realistic world simulator.<n>It generates egocentric videos that are strictly aligned with the real scene human motion of the user captured by an exocentric camera.
  arXiv  Detail & Related papers  (2025-06-11T17:59:53Z)
• EgoExo-Gen: Ego-centric Video Prediction by Watching Exo-centric Videos [49.24266108952835]
  Given an exo-centric video, the first frame of the corresponding ego-centric video, and textual instructions, the goal is to generate futur frames of the ego-centric video.<n>EgoExo-Gen explicitly models the hand-object dynamics for cross-view video prediction.
  arXiv  Detail & Related papers  (2025-04-16T03:12:39Z)
• Unlocking Exocentric Video-Language Data for Egocentric Video Representation Learning [80.37314291927889]
  We present EMBED, a method designed to transform exocentric video-language data for egocentric video representation learning. Egocentric videos predominantly feature close-up hand-object interactions, whereas exocentric videos offer a broader perspective on human activities. By applying both vision and language style transfer, our framework creates a new egocentric dataset.
  arXiv  Detail & Related papers  (2024-08-07T06:10:45Z)
• EgoExo-Fitness: Towards Egocentric and Exocentric Full-Body Action Understanding [27.881857222850083]
  EgoExo-Fitness is a new full-body action understanding dataset. It features fitness sequence videos recorded from synchronized egocentric and fixed exocentric cameras. EgoExo-Fitness provides new resources to study egocentric and exocentric full-body action understanding.
  arXiv  Detail & Related papers  (2024-06-13T07:28:45Z)
• EgoGen: An Egocentric Synthetic Data Generator [53.32942235801499]
  EgoGen is a new synthetic data generator that can produce accurate and rich ground-truth training data for egocentric perception tasks. At the heart of EgoGen is a novel human motion synthesis model that directly leverages egocentric visual inputs of a virtual human to sense the 3D environment. We demonstrate EgoGen's efficacy in three tasks: mapping and localization for head-mounted cameras, egocentric camera tracking, and human mesh recovery from egocentric views.
  arXiv  Detail & Related papers  (2024-01-16T18:55:22Z)
• Ego-Body Pose Estimation via Ego-Head Pose Estimation [22.08240141115053]
  Estimating 3D human motion from an egocentric video sequence plays a critical role in human behavior understanding and has various applications in VR/AR. We propose a new method, Ego-Body Pose Estimation via Ego-Head Pose Estimation (EgoEgo), which decomposes the problem into two stages, connected by the head motion as an intermediate representation. This disentanglement of head and body pose eliminates the need for training datasets with paired egocentric videos and 3D human motion.
  arXiv  Detail & Related papers  (2022-12-09T02:25:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.