LT3SD: Latent Trees for 3D Scene Diffusion

• URL: http://arxiv.org/abs/2409.08215v1
• Date: Thu, 12 Sep 2024 16:55:51 GMT
• Title: LT3SD: Latent Trees for 3D Scene Diffusion
• Authors: Quan Meng, Lei Li, Matthias Nießner, Angela Dai,
• Abstract summary: We present LT3SD, a novel latent diffusion model for large-scale 3D scene generation. We demonstrate the efficacy and benefits of LT3SD for large-scale, high-quality unconditional 3D scene generation.
• Score: 71.91446143124648
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present LT3SD, a novel latent diffusion model for large-scale 3D scene generation. Recent advances in diffusion models have shown impressive results in 3D object generation, but are limited in spatial extent and quality when extended to 3D scenes. To generate complex and diverse 3D scene structures, we introduce a latent tree representation to effectively encode both lower-frequency geometry and higher-frequency detail in a coarse-to-fine hierarchy. We can then learn a generative diffusion process in this latent 3D scene space, modeling the latent components of a scene at each resolution level. To synthesize large-scale scenes with varying sizes, we train our diffusion model on scene patches and synthesize arbitrary-sized output 3D scenes through shared diffusion generation across multiple scene patches. Through extensive experiments, we demonstrate the efficacy and benefits of LT3SD for large-scale, high-quality unconditional 3D scene generation and for probabilistic completion for partial scene observations.

Related papers

• L3DG: Latent 3D Gaussian Diffusion [74.36431175937285]
  L3DG is the first approach for generative 3D modeling of 3D Gaussians through a latent 3D Gaussian diffusion formulation. We employ a sparse convolutional architecture to efficiently operate on room-scale scenes. By leveraging the 3D Gaussian representation, the generated scenes can be rendered from arbitrary viewpoints in real-time.
  arXiv  Detail & Related papers  (2024-10-17T13:19:32Z)
• LN3Diff: Scalable Latent Neural Fields Diffusion for Speedy 3D Generation [73.36690511083894]
  This paper introduces a novel framework called LN3Diff to address a unified 3D diffusion pipeline. Our approach harnesses a 3D-aware architecture and variational autoencoder to encode the input image into a structured, compact, and 3D latent space. It achieves state-of-the-art performance on ShapeNet for 3D generation and demonstrates superior performance in monocular 3D reconstruction and conditional 3D generation.
  arXiv  Detail & Related papers  (2024-03-18T17:54:34Z)
• 3D-SceneDreamer: Text-Driven 3D-Consistent Scene Generation [51.64796781728106]
  We propose a generative refinement network to synthesize new contents with higher quality by exploiting the natural image prior to 2D diffusion model and the global 3D information of the current scene. Our approach supports wide variety of scene generation and arbitrary camera trajectories with improved visual quality and 3D consistency.
  arXiv  Detail & Related papers  (2024-03-14T14:31:22Z)
• BerfScene: Bev-conditioned Equivariant Radiance Fields for Infinite 3D Scene Generation [96.58789785954409]
  We propose a practical and efficient 3D representation that incorporates an equivariant radiance field with the guidance of a bird's-eye view map. We produce large-scale, even infinite-scale, 3D scenes via synthesizing local scenes and then stitching them with smooth consistency.
  arXiv  Detail & Related papers  (2023-12-04T18:56:10Z)
• Pyramid Diffusion for Fine 3D Large Scene Generation [56.00726092690535]
  Diffusion models have shown remarkable results in generating 2D images and small-scale 3D objects. Their application to the synthesis of large-scale 3D scenes has been rarely explored. We introduce a framework, the Pyramid Discrete Diffusion model (PDD), which employs scale-varied diffusion models to progressively generate high-quality outdoor scenes.
  arXiv  Detail & Related papers  (2023-11-20T11:24:21Z)
• 3inGAN: Learning a 3D Generative Model from Images of a Self-similar Scene [34.2144933185175]
  3inGAN is an unconditional 3D generative model trained from 2D images of a single self-similar 3D scene. We show results on semi-stochastic scenes of varying scale and complexity, obtained from real and synthetic sources.
  arXiv  Detail & Related papers  (2022-11-27T18:03:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.