Generation Models Know Space: Unleashing Implicit 3D Priors for Scene Understanding

• URL: http://arxiv.org/abs/2603.19235v1
• Date: Thu, 19 Mar 2026 17:59:58 GMT
• Title: Generation Models Know Space: Unleashing Implicit 3D Priors for Scene Understanding
• Abstract summary: We propose a paradigm shift by leveraging the implicit spatial prior within large-scale generation models.<n>We introduce GeneEGA-3D (Video Extracted Generative Awareness), a plug-and-play framework that repurposes a pre-trained video diffusion model as a Latent World Simulator.
• Score: 50.098085774845195
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: While Multimodal Large Language Models demonstrate impressive semantic capabilities, they often suffer from spatial blindness, struggling with fine-grained geometric reasoning and physical dynamics. Existing solutions typically rely on explicit 3D modalities or complex geometric scaffolding, which are limited by data scarcity and generalization challenges. In this work, we propose a paradigm shift by leveraging the implicit spatial prior within large-scale video generation models. We posit that to synthesize temporally coherent videos, these models inherently learn robust 3D structural priors and physical laws. We introduce VEGA-3D (Video Extracted Generative Awareness), a plug-and-play framework that repurposes a pre-trained video diffusion model as a Latent World Simulator. By extracting spatiotemporal features from intermediate noise levels and integrating them with semantic representations via a token-level adaptive gated fusion mechanism, we enrich MLLMs with dense geometric cues without explicit 3D supervision. Extensive experiments across 3D scene understanding, spatial reasoning, and embodied manipulation benchmarks demonstrate that our method outperforms state-of-the-art baselines, validating that generative priors provide a scalable foundation for physical-world understanding. Code is publicly available at https://github.com/H-EmbodVis/VEGA-3D.

Related papers

• Stream3D-VLM: Online 3D Spatial Understanding with Incremental Geometry Priors [54.51965288351254]
  We present an online 3D vision-language model that enables real-time spatial understanding from streaming video.<n>Our approach adopts an autoregressive streaming control modeling based on the LLM's next-token prediction to learn when to respond.<n>To address the scarcity of streaming 3D-language data, we further develop a scalable data generation pipeline.
  arXiv  Detail & Related papers  (2026-06-05T04:16:24Z)
• 3D-IDE: 3D Implicit Depth Emergent [45.72771473431863]
  3D-Implicit Depth Emergence is a method that reframes 3D perception as an emergent property derived from geometric self-vision.<n>Our method enables 3D perception to emerge implicitly, disentangling features in dense regions.<n>Our approach achieves a 55% reduction in inference latency while maintaining strong performance across diverse downstream tasks.
  arXiv  Detail & Related papers  (2026-03-28T00:54:19Z)
• Know3D: Prompting 3D Generation with Knowledge from Vision-Language Models [39.48274812723658]
  Know3D is a novel framework that incorporates rich knowledge from large language models into 3D generative processes.<n>We successfully bridge the gap between abstract instructions and the geometric reconstruction of unobserved regions.
  arXiv  Detail & Related papers  (2026-03-24T04:18:29Z)
• 3DreamBooth: High-Fidelity 3D Subject-Driven Video Generation Model [18.407923160013404]
  We introduce a novel framework for 3D-aware video customization, comprising 3DreamBooth and 3Dapter.<n>3DreamBooth decouples spatial geometry from temporal motion through a 1-frame optimization paradigm.<n>To enhance fine-grained textures and accelerate convergence, we incorporate 3Dapter, a visual conditioning module.
  arXiv  Detail & Related papers  (2026-03-19T06:14:05Z)
• GAP-MLLM: Geometry-Aligned Pre-training for Activating 3D Spatial Perception in Multimodal Large Language Models [70.61152292499737]
  We argue that this gap does not arise from insufficient geometric priors, but from a misalignment in the training paradigm.<n>Existing approaches typically resort to naive feature concatenation and optimize directly for downstream tasks without geometry-specific supervision.<n>We propose GAP-MLLM, a Geometry-Aligned Pre-training paradigm that explicitly activates structural perception before downstream adaptation.
  arXiv  Detail & Related papers  (2026-03-17T12:43:48Z)
• Scalable Adaptation of 3D Geometric Foundation Models via Weak Supervision from Internet Video [76.32954467706581]
  We propose SAGE, a framework for Scalable Adaptation of GEometric foundation models from raw video streams.<n>We use a hierarchical mining pipeline to transform videos into training trajectories and hybrid supervision.<n>Experiments show that SAGE significantly enhances zero-shot generalization, reducing Chamfer Distance by 20-42% on unseen benchmarks.
  arXiv  Detail & Related papers  (2026-02-08T09:53:21Z)
• AlignGS: Aligning Geometry and Semantics for Robust Indoor Reconstruction from Sparse Views [18.361136390711415]
  The demand for semantically rich 3D models of indoor scenes is rapidly growing, driven by applications in augmented reality, virtual reality, and robotics.<n>Existing methods often treat semantics as a passive feature painted on an already-formed, and potentially flawed, geometry.<n>This paper introduces AlignGS, a novel framework that actualizes this vision by pioneering a synergistic, end-to-end optimization of geometry and semantics.
  arXiv  Detail & Related papers  (2025-10-09T06:30:20Z)
• FantasyWorld: Geometry-Consistent World Modeling via Unified Video and 3D Prediction [13.098585993121722]
  We present FantasyWorld, a geometry-enhanced framework that augments frozen video foundation models with a trainable geometric branch.<n>Our approach introduces cross-branch supervision, where geometry cues guide video generation and video priors regularize 3D prediction.<n>Experiments show that FantasyWorld effectively bridges video imagination and 3D perception, outperforming recent geometry-consistent baselines in multi-view coherence and style consistency.
  arXiv  Detail & Related papers  (2025-09-25T22:24:23Z)
• UniUGG: Unified 3D Understanding and Generation via Geometric-Semantic Encoding [65.60549881706959]
  We introduce UniUGG, the first unified understanding and generation framework for 3D modalities.<n>Our framework employs an LLM to comprehend and decode sentences and 3D representations.<n>We propose a spatial decoder leveraging a latent diffusion model to generate high-quality 3D representations.
  arXiv  Detail & Related papers  (2025-08-16T07:27:31Z)
• Agentic 3D Scene Generation with Spatially Contextualized VLMs [67.31920821192323]
  We introduce a new paradigm that enables vision-language models to generate, understand, and edit complex 3D environments.<n>We develop an agentic 3D scene generation pipeline in which the VLM iteratively reads from and updates the spatial context.<n>Results show that our framework can handle diverse and challenging inputs, achieving a level of generalization not observed in prior work.
  arXiv  Detail & Related papers  (2025-05-26T15:28:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.