SpatialCrafter: Single Image World Modeling with Generative 3D Proxies

• URL: http://arxiv.org/abs/2608.27073v2
• Date: Fri, 28 Aug 2026 15:48:43 GMT
• Title: SpatialCrafter: Single Image World Modeling with Generative 3D Proxies
• Abstract summary: Video diffusion model (VDM) commonly rely on conditioning incomplete signals such as sparse point clouds or 2D panoramas.<n>We present a novel two-stage framework that addresses these issues by introducing a global 3D proxy for high-fidelity image-to-scene generation.<n>For proxy generation, we propose a Point-anchored Sparse Structure(PaSS) Flow module that predicts a spatially aligned and geometrically consistent 3D proxy.<n>For appearance refinement, we re-frame the VDM as a Generative Deferred Refiner which synthesizes high-frequency photorealistic details upon proxy-defined scene geometry
• Score: 41.996454262653735
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Explorable image-to-scene generation is essential for applications in gaming, robotics, and virtual reality. Existing methods based on video diffusion model (VDM) commonly rely on incomplete conditioning signals such as sparse point clouds or 2D panoramas, leading to stochastic hallucinations, long-term drifts and suboptimal 3D consistency. We present SpatialCrafter, a novel two-stage framework that addresses these issues by introducing a global 3D proxy for high-fidelity image-to-scene generation. Specifically, we decompose the generation process into global proxy generation and appearance refinement. For proxy generation, we propose a Point-anchored Sparse Structure~(PaSS) Flow module that predicts a spatially aligned and geometrically consistent 3D proxy. For appearance refinement, we re-frame the VDM as a Generative Deferred Refiner which synthesizes high-frequency photorealistic details upon proxy-defined scene geometry. To better integrate the proxy with the pre-trained VDM, we introduce Parallel Geometry Injection and Proxy-Aware Corruption training strategies, which improve robustness to proxy artifacts without disrupting the pretrained generative manifold. Furthermore, as no suitable dataset exists for this explorable scene generation task, we construct a new large-scale dataset of 115K scenes. To the best of our knowledge, it is the first hybrid dataset for image-to-scene generation. Extensive experiments on both synthetic and real-world datasets show that SpatialCrafter outperforms state-of-the-art methods, mitigates long-term drift, and remains robust and consistent under rapid camera motion and extreme viewpoint changes. Our project page: \href{https://fangchuan.github.io/SpatialCrafter/}{fangchuan.github.io/SpatialCrafter/}

Related papers

• Lyra 2.0: Explorable Generative 3D Worlds [77.45279013687427]
  We present Lyra 2.0, a framework for generating persistent, explorable 3D worlds at scale.<n>To address spatial forgetting, we maintain per-frame 3D geometry and use it solely for information routing.<n>We train with self-augmented histories that expose the model to its own degraded outputs, teaching it to correct drift rather than propagate it.
  arXiv  Detail & Related papers  (2026-04-14T17:59:44Z)
• FreeScale: Scaling 3D Scenes via Certainty-Aware Free-View Generation [75.74617373156902]
  FreeScale is a framework that transforms limited real-world image sequences into a scalable source of high-quality training data.<n>We show FreeScale's effectiveness by scaling up the training of feedforward NVS models, achieving a notable gain of 2.7 dB in PSNR.<n>Our work provides a practical and powerful data generation engine to overcome a fundamental bottleneck in 3D vision.
  arXiv  Detail & Related papers  (2026-04-12T08:00:53Z)
• WorldFlow3D: Flowing Through 3D Distributions for Unbounded World Generation [33.369156362316055]
  Unbounded 3D world generation is emerging as a foundational task for scene modeling in computer vision, graphics, and robotics.<n>We present WorldFlow3D, a novel method capable of generating unbounded 3D worlds.
  arXiv  Detail & Related papers  (2026-03-31T00:08:17Z)
• Native and Compact Structured Latents for 3D Generation [44.84214149555589]
  We present an approach for learning a structured latent representation from native 3D data to address this challenge.<n>At its core is a new sparse voxel structure called O-Voxel, an omni-voxel representation that encodes both geometry and appearance.<n>Based on O-Voxel, we design a Sparse Compression VAE which provides a high spatial compression rate and a compact latent space.
  arXiv  Detail & Related papers  (2025-12-16T18:58:28Z)
• WorldGrow: Generating Infinite 3D World [75.81531067447203]
  We tackle the challenge of generating the infinitely extendable 3D world -- large, continuous environments with coherent geometry and realistic appearance.<n>We propose WorldGrow, a hierarchical framework for unbounded 3D scene synthesis.<n>Our method features three core components: (1) a data curation pipeline that extracts high-quality scene blocks for training, making the 3D structured latent representations suitable for scene generation; (2) a 3D block inpainting mechanism that enables context-aware scene extension; and (3) a coarse-to-fine generation strategy that ensures both global layout plausibility and local geometric/textural fidelity.
  arXiv  Detail & Related papers  (2025-10-24T17:39:52Z)
• Topology Sculptor, Shape Refiner: Discrete Diffusion Model for High-Fidelity 3D Meshes Generation [14.55646181682844]
  Topology Sculptor, Shape Refiner (TSSR) is a novel method for generating high-quality, artist-style 3D meshes.<n>We leverage this parallel generation capability through three key innovations.<n> experiments on complex datasets demonstrate that TSSR generates high-quality 3D artist-style meshes.
  arXiv  Detail & Related papers  (2025-10-24T08:51:48Z)
• OracleGS: Grounding Generative Priors for Sparse-View Gaussian Splatting [78.70702961852119]
  OracleGS reconciles generative completeness with regressive fidelity for sparse view Gaussian Splatting.<n>Our approach conditions the powerful generative prior on multi-view geometric evidence, filtering hallucinatory artifacts while preserving plausible completions in under-constrained regions.
  arXiv  Detail & Related papers  (2025-09-27T11:19:32Z)
• 3D-SceneDreamer: Text-Driven 3D-Consistent Scene Generation [51.64796781728106]
  We propose a generative refinement network to synthesize new contents with higher quality by exploiting the natural image prior to 2D diffusion model and the global 3D information of the current scene. Our approach supports wide variety of scene generation and arbitrary camera trajectories with improved visual quality and 3D consistency.
  arXiv  Detail & Related papers  (2024-03-14T14:31:22Z)
• OSTeC: One-Shot Texture Completion [86.23018402732748]
  We propose an unsupervised approach for one-shot 3D facial texture completion. The proposed approach rotates an input image in 3D and fill-in the unseen regions by reconstructing the rotated image in a 2D face generator. We frontalize the target image by projecting the completed texture into the generator.
  arXiv  Detail & Related papers  (2020-12-30T23:53:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.