EgoForge: Goal-Directed Egocentric World Simulator

• URL: http://arxiv.org/abs/2603.20169v1
• Date: Fri, 20 Mar 2026 17:46:55 GMT
• Title: EgoForge: Goal-Directed Egocentric World Simulator
• Abstract summary: EgoForge is a goal-directed world simulator that generates coherent, first-person video rollouts from minimal static inputs.<n>VideoDiffusionNFT is a trajectory-level reward-guided refinement that optimize goal completion, temporal causality, scene consistency, and perceptual fidelity during diffusion sampling.
• Score: 26.712565464146937
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Generative world models have shown promise for simulating dynamic environments, yet egocentric video remains challenging due to rapid viewpoint changes, frequent hand-object interactions, and goal-directed procedures whose evolution depends on latent human intent. Existing approaches either focus on hand-centric instructional synthesis with limited scene evolution, perform static view translation without modeling action dynamics, or rely on dense supervision, such as camera trajectories, long video prefixes, synchronized multicamera capture, etc. In this work, we introduce EgoForge, an egocentric goal-directed world simulator that generates coherent, first-person video rollouts from minimal static inputs: a single egocentric image, a high-level instruction, and an optional auxiliary exocentric view. To improve intent alignment and temporal consistency, we propose VideoDiffusionNFT, a trajectory-level reward-guided refinement that optimizes goal completion, temporal causality, scene consistency, and perceptual fidelity during diffusion sampling. Extensive experiments show EgoForge achieves consistent gains in semantic alignment, geometric stability, and motion fidelity over strong baselines, and robust performance in real-world smart-glasses experiments.

Related papers

• AnchorWorld: Embodied Egocentric World Simulation with View-based Evolution Customization [79.26935895370191]
  AnchorWorld is a framework that advances egocentric simulation through enhanced interaction integrity.<n> auxiliary training supervision incorporates viewpoints decoupled from the agent's first-person sensorium.<n>We propose a simple yet effective mechanism for customizing self-evolving worlds.
  arXiv  Detail & Related papers  (2026-06-05T14:43:13Z)
• EgoMotion: Hierarchical Reasoning and Diffusion for Egocentric Vision-Language Motion Generation [74.07852250099559]
  We investigate Egocentric Vision-Language (Ego-VL) motion generation.<n>This task requires synthesizing 3D human motion conditioned jointly on first-person visual observations and natural language instructions.<n>We propose a hierarchical generative framework textbfEgoMotion to address this challenge.
  arXiv  Detail & Related papers  (2026-04-21T05:31:06Z)
• EgoFlow: Gradient-Guided Flow Matching for Egocentric 6DoF Object Motion Generation [47.32597153743819]
  We present EgoFlow, a flow-matching framework that synthesizes realistic and physically plausible trajectories conditioned on multimodal egocentric observations.<n>Our results highlight the promise of flow-based generative modeling for scalable and physically grounded egocentric motion understanding.
  arXiv  Detail & Related papers  (2026-04-01T21:43:57Z)
• Hand2World: Autoregressive Egocentric Interaction Generation via Free-Space Hand Gestures [33.2764643227486]
  Egocentric interactive world models are essential for augmented reality and embodied AI, where visual generation must respond to user input with low latency, geometric consistency, and long-term stability.<n>We study egocentric interaction generation from a single scene image under free-space hand gestures, aiming to synthesize photorealistic videos in which hands enter the scene, interact with objects, and induce plausible world dynamics under head motion.<n>This setting introduces fundamental challenges, including distribution shift between free-space gestures and contact-heavy training data, ambiguity between hand motion and camera motion in monocular views, and the need for arbitrary-length video generation.
  arXiv  Detail & Related papers  (2026-02-10T09:51:07Z)
• Walk through Paintings: Egocentric World Models from Internet Priors [65.30611174953958]
  We present the Egocentric World Model (EgoWM), a simple, architecture-agnostic method that transforms any pretrained video diffusion model into an action-conditioned world model.<n>Rather than training from scratch, we repurpose the rich world priors of Internet-scale video models and inject motor commands through lightweight conditioning layers.<n>Our approach scales naturally across embodiments and action spaces, ranging from 3-DoF mobile robots to 25-DoF humanoids.
  arXiv  Detail & Related papers  (2026-01-21T18:59:32Z)
• EgoGrasp: World-Space Hand-Object Interaction Estimation from Egocentric Videos [25.047225764745978]
  We propose EgoGrasp, the first method to reconstruct world-space hand-object interactions (W-HOI) from egocentric monocular videos with dynamic cameras in the wild.<n>In experiments, we prove our method achieving state-of-the-art performance in W-HOI reconstruction.
  arXiv  Detail & Related papers  (2026-01-03T03:08:48Z)
• TeleWorld: Towards Dynamic Multimodal Synthesis with a 4D World Model [53.555353366322464]
  We present TeleWorld, a real-time multimodal 4D world modeling framework that unifies video generation, dynamic scene reconstruction, and long-term world memory within a closed-loop system.<n>Our approach achieves seamless integration of dynamic object modeling and static scene representation within a unified 4D framework, advancing world models toward practical, interactive, and computationally accessible synthesis systems.
  arXiv  Detail & Related papers  (2025-12-31T18:31:46Z)
• Hand-Aware Egocentric Motion Reconstruction with Sequence-Level Context [17.735273173582716]
  We present HaMoS, the first hand-aware, sequence-level diffusion framework that directly conditions on both head trajectory and intermittently visible hand cues.<n>We also demonstrate that sequence-level contexts such as body shape and field-of-view are crucial for accurate motion reconstruction.
  arXiv  Detail & Related papers  (2025-12-22T11:26:41Z)
• PlayerOne: Egocentric World Simulator [73.88786358213694]
  PlayerOne is the first egocentric realistic world simulator.<n>It generates egocentric videos that are strictly aligned with the real scene human motion of the user captured by an exocentric camera.
  arXiv  Detail & Related papers  (2025-06-11T17:59:53Z)
• EgoM2P: Egocentric Multimodal Multitask Pretraining [55.259234688003545]
  Building large-scale egocentric multimodal and multitask models presents unique challenges.<n> EgoM2P is a masked modeling framework that learns from temporally-aware multimodal tokens to train a large, general-purpose model for egocentric 4D understanding.<n>We will fully open-source EgoM2P to support the community and advance egocentric vision research.
  arXiv  Detail & Related papers  (2025-06-09T15:59:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.