EgoSim: Egocentric World Simulator for Embodied Interaction Generation

• URL: http://arxiv.org/abs/2604.01001v1
• Date: Wed, 01 Apr 2026 15:00:46 GMT
• Title: EgoSim: Egocentric World Simulator for Embodied Interaction Generation
• Abstract summary: EgoSim is a closed-loop egocentric world simulator that generates spatially consistent interaction videos.<n>It persistently updates the underlying 3D scene state for continuous simulation.<n>EgoSim significantly outperforms existing methods in terms of visual quality, spatial consistency, and generalization.
• Score: 93.11209644808783
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We introduce EgoSim, a closed-loop egocentric world simulator that generates spatially consistent interaction videos and persistently updates the underlying 3D scene state for continuous simulation. Existing egocentric simulators either lack explicit 3D grounding, causing structural drift under viewpoint changes, or treat the scene as static, failing to update world states across multi-stage interactions. EgoSim addresses both limitations by modeling 3D scenes as updatable world states. We generate embodiment interactions via a Geometry-action-aware Observation Simulation model, with spatial consistency from an Interaction-aware State Updating module. To overcome the critical data bottleneck posed by the difficulty in acquiring densely aligned scene-interaction training pairs, we design a scalable pipeline that extracts static point clouds, camera trajectories, and embodiment actions from in-the-wild large-scale monocular egocentric videos. We further introduce EgoCap, a capture system that enables low-cost real-world data collection with uncalibrated smartphones. Extensive experiments demonstrate that EgoSim significantly outperforms existing methods in terms of visual quality, spatial consistency, and generalization to complex scenes and in-the-wild dexterous interactions, while supporting cross-embodiment transfer to robotic manipulation. Codes and datasets will be open soon. The project page is at egosimulator.github.io.

Related papers

• EgoTrack3D: A Modular Framework for Egocentric 3D Object Tracking [48.532819029347785]
  We introduce EgoTrack3D, a modular framework that reconstructs and maintains a dynamic 3D scene representation directly from egocentric RGB video.<n> EgoTrack3D maintains accurate representations over time, achieving an 11% improvement in percentage of correct locations.<n>We replace dense depth maps with sparse 3D bounding box estimation and integrate interaction-guided dynamic association.
  arXiv  Detail & Related papers  (2026-08-08T08:53:54Z)
• EgoInteract: Synthetic Egocentric Videos Generation for Interaction Understanding and Anticipation [45.01838097419948]
  We introduce a controllable simulator for egocentric video generation designed to model fine-grained egocentric interactions and their temporal dynamics.<n>We generate a synthetic egocentric video dataset with dense spatial and temporal annotations for temporal action segmentation, next-active object detection, interaction anticipation, and hand-object interaction detection.<n>Results show consistent improvements over strong baselines across tasks and datasets, demonstrating the effectiveness and transferability of our simulation-based approach.
  arXiv  Detail & Related papers  (2026-05-18T10:58:56Z)
• Ego2World: Compiling Egocentric Cooking Videos into Executable Worlds for Belief-State Planning [55.43343782036886]
  We introduce Ego2World, an executable benchmark that turns egocentric cooking videos into executable symbolic worlds governed by graph-transition rules.<n>During evaluation, the simulator maintains a hidden world graph, while the agent plans over its own partial belief graph using only local observations and execution feedback.<n>This separation forces agents to update memory and replan without observing the true world state.
  arXiv  Detail & Related papers  (2026-05-13T10:53:15Z)
• EgoFlow: Gradient-Guided Flow Matching for Egocentric 6DoF Object Motion Generation [47.32597153743819]
  We present EgoFlow, a flow-matching framework that synthesizes realistic and physically plausible trajectories conditioned on multimodal egocentric observations.<n>Our results highlight the promise of flow-based generative modeling for scalable and physically grounded egocentric motion understanding.
  arXiv  Detail & Related papers  (2026-04-01T21:43:57Z)
• Egocentric World Model for Photorealistic Hand-Object Interaction Synthesis [25.153233500074194]
  EgoHOI is an egocentric HOI world model that breaks away from this shortcut to simulate photorealistic, contact-consistent interactions from action signals alone.<n>EgoHOI distills geometric and kinematic priors from 3D estimates into physics-informed embeddings.<n>Experiments on the HOT3D dataset demonstrate consistent gains over strong baselines, and ablations validate the effectiveness of our design.
  arXiv  Detail & Related papers  (2026-03-13T21:46:17Z)
• EgoReAct: Egocentric Video-Driven 3D Human Reaction Generation [84.37917777533963]
  We present EgoReAct, the first framework that generates 3D-aligned human reaction motions from egocentric video streams in real-time.<n>EgoReAct achieves remarkably higher realism, spatial consistency, and generation efficiency compared with prior methods.
  arXiv  Detail & Related papers  (2025-12-28T06:44:05Z)
• PlayerOne: Egocentric World Simulator [73.88786358213694]
  PlayerOne is the first egocentric realistic world simulator.<n>It generates egocentric videos that are strictly aligned with the real scene human motion of the user captured by an exocentric camera.
  arXiv  Detail & Related papers  (2025-06-11T17:59:53Z)
• EgoGen: An Egocentric Synthetic Data Generator [53.32942235801499]
  EgoGen is a new synthetic data generator that can produce accurate and rich ground-truth training data for egocentric perception tasks. At the heart of EgoGen is a novel human motion synthesis model that directly leverages egocentric visual inputs of a virtual human to sense the 3D environment. We demonstrate EgoGen's efficacy in three tasks: mapping and localization for head-mounted cameras, egocentric camera tracking, and human mesh recovery from egocentric views.
  arXiv  Detail & Related papers  (2024-01-16T18:55:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.