Sparse auto-regressive modeling for scene generation from multi-view images

• URL: http://arxiv.org/abs/2609.03931v1
• Date: Thu, 03 Sep 2026 14:41:34 GMT
• Title: Sparse auto-regressive modeling for scene generation from multi-view images
• Abstract summary: We introduce SPAR3S, a sparse voxel-aligned 3D latent generative model for conditional scene completion.<n>Our key insight is to formulate 3D scene generation in a structured, compact, voxel-aligned 3D latent space.<n>We demonstrate the effectiveness of our method on synthetic indoor scenes, achieving higher novel-view quality than prior work.
• Score: 20.272184280044083
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Generating complete 3D scenes from sparse, unconstrained views is a fundamental challenge in 3D vision which requires reasoning beyond observed content while remaining computationally tractable. Existing feed-forward reconstruction methods are inherently limited to content visible in the input images, while 3D generative modeling is hindered by the high computational cost of dense volumetric representations and the scarcity of large-scale 3D supervision. We introduce SPAR3S, a sparse voxel-aligned 3D latent generative model for conditional scene completion without requiring ground-truth 3D data for supervision. Our key insight is to formulate 3D scene generation in a structured, compact, voxel-aligned 3D latent space where only occupied voxels are represented. We learn this sparse latent space directly from multi-view images using photometric supervision via differentiable 3D Gaussian Splatting. Given a partial set of observed voxels encoded from sparse input views, scene completion reduces to predicting the missing latent tokens and their spatial support within the voxel grid. To this end, we train a masked autoregressive transformer that jointly models voxel occupancy and latent token values, enabling efficient and spatially consistent generation of unseen regions. We demonstrate the effectiveness of our method on synthetic indoor scenes, achieving higher novel-view quality than prior work. We further validate its generalization on RealEstate10k, highlighting its applicability to real-world data.

Related papers

• SEM-ROVER: Semantic Voxel-Guided Diffusion for Large-Scale Driving Scene Generation [7.655860434125127]
  We propose a 3D generative framework based on $$-Voxfield grid, a discrete representation where each occupied voxel stores a fixed number of colorized surface samples.<n>To generate this representation, we train a semantic-conditioned diffusion model that operates on local voxel neighborhoods.<n>We scale to large scenes via progressive spatial outpainting over overlapping regions. Finally, we render the generated $$-Voxfield grid with a deferred rendering module to obtain photorealistic images.
  arXiv  Detail & Related papers  (2026-04-07T17:24:29Z)
• Seen2Scene: Completing Realistic 3D Scenes with Visibility-Guided Flow [80.05357398241574]
  We present Seen2Scene, the first flow matching-based approach that trains on incomplete, real-world 3D scans for scene completion and generation.<n>By learning directly from real-world, incomplete 3D scans, Seen2Scene enables realistic 3D scene completion for complex, cluttered real environments.
  arXiv  Detail & Related papers  (2026-03-30T15:08:38Z)
• SPATIALGEN: Layout-guided 3D Indoor Scene Generation [37.30623176278608]
  We present SpatialGen, a novel multi-view multi-modal diffusion model that generates realistic and semantically consistent 3D indoor scenes.<n>Given a 3D layout and a reference image, our model synthesizes appearance (color image), geometry (scene coordinate map), and semantic (semantic segmentation map) from arbitrary viewpoints.<n>We are open-sourcing our data and models to empower the community and advance the field of indoor scene understanding and generation.
  arXiv  Detail & Related papers  (2025-09-18T14:12:32Z)
• UniForward: Unified 3D Scene and Semantic Field Reconstruction via Feed-Forward Gaussian Splatting from Only Sparse-View Images [43.40816438003861]
  We propose a feed-forward model that unifies 3D scene and semantic field reconstruction.<n>Our UniForward can reconstruct 3D scenes and the corresponding semantic fields in real time from only sparse-view images.<n> Experiments on novel view synthesis and novel view segmentation demonstrate that our method achieves state-of-the-art performances.
  arXiv  Detail & Related papers  (2025-06-11T04:01:21Z)
• EVolSplat: Efficient Volume-based Gaussian Splatting for Urban View Synthesis [61.1662426227688]
  Existing NeRF and 3DGS-based methods show promising results in achieving photorealistic renderings but require slow, per-scene optimization.<n>We introduce EVolSplat, an efficient 3D Gaussian Splatting model for urban scenes that works in a feed-forward manner.
  arXiv  Detail & Related papers  (2025-03-26T02:47:27Z)
• SCube: Instant Large-Scale Scene Reconstruction using VoxSplats [55.383993296042526]
  We present SCube, a novel method for reconstructing large-scale 3D scenes (geometry, appearance, and semantics) from a sparse set of posed images. Our method encodes reconstructed scenes using a novel representation VoxSplat, which is a set of 3D Gaussians supported on a high-resolution sparse-voxel scaffold.
  arXiv  Detail & Related papers  (2024-10-26T00:52:46Z)
• Denoising Diffusion via Image-Based Rendering [54.20828696348574]
  We introduce the first diffusion model able to perform fast, detailed reconstruction and generation of real-world 3D scenes. First, we introduce a new neural scene representation, IB-planes, that can efficiently and accurately represent large 3D scenes. Second, we propose a denoising-diffusion framework to learn a prior over this novel 3D scene representation, using only 2D images.
  arXiv  Detail & Related papers  (2024-02-05T19:00:45Z)
• NeuralField-LDM: Scene Generation with Hierarchical Latent Diffusion Models [85.20004959780132]
  We introduce NeuralField-LDM, a generative model capable of synthesizing complex 3D environments. We show how NeuralField-LDM can be used for a variety of 3D content creation applications, including conditional scene generation, scene inpainting and scene style manipulation.
  arXiv  Detail & Related papers  (2023-04-19T16:13:21Z)
• LiP-Flow: Learning Inference-time Priors for Codec Avatars via Normalizing Flows in Latent Space [90.74976459491303]
  We introduce a prior model that is conditioned on the runtime inputs and tie this prior space to the 3D face model via a normalizing flow in the latent space. A normalizing flow bridges the two representation spaces and transforms latent samples from one domain to another, allowing us to define a latent likelihood objective. We show that our approach leads to an expressive and effective prior, capturing facial dynamics and subtle expressions better.
  arXiv  Detail & Related papers  (2022-03-15T13:22:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.