Grounded-Exo2Ego: Structured Semantic Grounding for Robust Exocentric-to-Egocentric Video Generation

• URL: http://arxiv.org/abs/2608.20534v1
• Date: Thu, 20 Aug 2026 19:57:04 GMT
• Title: Grounded-Exo2Ego: Structured Semantic Grounding for Robust Exocentric-to-Egocentric Video Generation
• Abstract summary: Generating egocentric video from a single exocentric video is an emerging and important topic for AR/VR and physical AI.<n>We present Grounded-Exo2Ego, a principled framework that addresses these challenges at both the architectural and data levels.
• Score: 25.65935430003468
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Generating egocentric video from a single exocentric video is an emerging and important topic for AR/VR and physical AI. Compared with conventional novel view synthesis, exo-to-ego generation is a significantly harder task because the standard geometric conditioning becomes highly unreliable under extreme view changes and large unobservable regions. We present Grounded-Exo2Ego, a principled framework that addresses these challenges at both the architectural and data levels. Architecturally, Grounded-Exo2Ego is a dual-branch video diffusion model that couples a geometric anchoring branch, which conditions the generation on the rendering of a 3D reconstruction, with a novel semantic grounding branch, which goes beyond the prevailing geometry-based approach and improves quality by synthesizing challenging regions based on object-level context. Additionally, we found that the overlooked issue of camera-reconstruction misalignment severely undermines exo-to-ego learning. We thus introduce a camera re-localization algorithm that resolves this issue and substantially improves quality across all metrics. We further develop a fully automated synthetic data engine that generates and renders rigged 3D characters in procedurally generated environments. Evaluation on the challenging EgoExo4D dataset shows that our method outperforms recent state-of-the-art approaches by large margins across all metrics. Detailed ablations validate improvements from each of our contributions at both the data and architectural level.

Related papers

• CGGS: Consistency-Augmented Geometric Gaussian Splatting for Ego-centric 3D Scene Generation [14.481547285312928]
  CGGS aims to enhance 3D-content-awareness and address geometric distortions in ego-centric scene generation.<n>Ego-centric Generator is proposed by fine-tuning a Multi-View Latent Diffusion Model with consistency-augmented loss.<n> layout Decorator leverages optical flow and point-track correspondence to estimate depth.<n> Geometric Refiner is proposed to enhance 3D Gaussian reconstruction via an entropy-based Mutual Information Depth Loss.
  arXiv  Detail & Related papers  (2026-07-04T11:06:41Z)
• Xiaomi Auto World Model: A Joint World Model Integrating Reconstruction and Generation for Autonomous Driving [51.90209659403234]
  This report presents a unified technical system addressing the two core capabilities of world models for autonomous driving.<n>For world representation, we propose WorldRec, a feed-forward reconstruction architecture driven by sparse scene queries.<n>For world generation, we propose WorldGen, a two-stage training framework of bidirectional pretraining followed by causal fine-tuning.
  arXiv  Detail & Related papers  (2026-05-18T09:46:16Z)
• World-R1: Reinforcing 3D Constraints for Text-to-Video Generation [36.18760315762835]
  We propose World-R1, a framework that aligns video generation with 3D constraints through reinforcement learning.<n>We optimize the model using feedback from pre-trained 3D foundation models and vision-language models.<n>Our approach significantly enhances 3D consistency while preserving the original visual quality of the foundation model.
  arXiv  Detail & Related papers  (2026-04-27T17:59:56Z)
• Bringing a Personal Point of View: Evaluating Dynamic 3D Gaussian Splatting for Egocentric Scene Reconstruction [17.903916530175533]
  Egocentric video provides a unique view into human perception and interaction.<n>Rapid camera motion and complex scene dynamics pose major challenges for 3D reconstruction from this perspective.<n>We evaluate dynamic monocular 3DGS models on egocentric and exocentric video using paired ego-exo recordings.
  arXiv  Detail & Related papers  (2026-04-26T16:58:19Z)
• OneWorld: Taming Scene Generation with 3D Unified Representation Autoencoder [90.8453349494245]
  We present OneWorld, a framework that performs diffusion directly within a coherent 3D representation space.<n>OneWorld generates high-quality 3D scenes with superior cross-view consistency compared to state-of-the-art 2D-based methods.
  arXiv  Detail & Related papers  (2026-03-17T03:43:37Z)
• RAYNOVA: Scale-Temporal Autoregressive World Modeling in Ray Space [51.441415833480505]
  RAYNOVA is a multiview world model for driving scenarios that employs a dual-causal autoregressive framework.<n>It constructs an isotropic-temporal representation across views, frames, and scales based on relative Plcker-ray positional encoding.
  arXiv  Detail & Related papers  (2026-02-24T08:41:40Z)
• Scalable Adaptation of 3D Geometric Foundation Models via Weak Supervision from Internet Video [76.32954467706581]
  We propose SAGE, a framework for Scalable Adaptation of GEometric foundation models from raw video streams.<n>We use a hierarchical mining pipeline to transform videos into training trajectories and hybrid supervision.<n>Experiments show that SAGE significantly enhances zero-shot generalization, reducing Chamfer Distance by 20-42% on unseen benchmarks.
  arXiv  Detail & Related papers  (2026-02-08T09:53:21Z)
• GeoDiff3D: Self-Supervised 3D Scene Generation with Geometry-Constrained 2D Diffusion Guidance [8.625308061265754]
  3D scene generation is a core technology for gaming, film/VFX, and VR/AR.<n>Existing methods broadly follow two paradigms - indirect 2D-to-3D reconstruction and direct 3D generation.<n>We propose GeoDiff3D, an efficient self-supervised framework that uses coarse geometry as a structural anchor and a geometry-constrained 2D diffusion model to provide texture-rich reference images.
  arXiv  Detail & Related papers  (2026-01-27T16:47:35Z)
• WorldGrow: Generating Infinite 3D World [75.81531067447203]
  We tackle the challenge of generating the infinitely extendable 3D world -- large, continuous environments with coherent geometry and realistic appearance.<n>We propose WorldGrow, a hierarchical framework for unbounded 3D scene synthesis.<n>Our method features three core components: (1) a data curation pipeline that extracts high-quality scene blocks for training, making the 3D structured latent representations suitable for scene generation; (2) a 3D block inpainting mechanism that enables context-aware scene extension; and (3) a coarse-to-fine generation strategy that ensures both global layout plausibility and local geometric/textural fidelity.
  arXiv  Detail & Related papers  (2025-10-24T17:39:52Z)
• HumanGenesis: Agent-Based Geometric and Generative Modeling for Synthetic Human Dynamics [60.737929335600015]
  We present textbfHumanGenesis, a framework that integrates geometric and generative modeling through four collaborative agents.<n>HumanGenesis achieves state-of-the-art performance on tasks including text-guided synthesis, video reenactment, and novel-pose generalization.
  arXiv  Detail & Related papers  (2025-08-13T14:50:19Z)
• GTR: Gaussian Splatting Tracking and Reconstruction of Unknown Objects Based on Appearance and Geometric Complexity [49.31257173003408]
  We present a novel method for 6-DoF object tracking and high-quality 3D reconstruction from monocular RGBD video.<n>Our approach demonstrates strong capabilities in recovering high-fidelity object meshes, setting a new standard for single-sensor 3D reconstruction in open-world environments.
  arXiv  Detail & Related papers  (2025-05-17T08:46:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.