DiffInDScene: Diffusion-based High-Quality 3D Indoor Scene Generation

• URL: http://arxiv.org/abs/2306.00519v4
• Date: Wed, 29 Nov 2023 03:55:48 GMT
• Title: DiffInDScene: Diffusion-based High-Quality 3D Indoor Scene Generation
• Abstract summary: DiffInDScene is a novel framework for tackling the problem of high-quality 3D indoor scene generation. Inspired by KinectFusion's incremental alignment and fusion of local TSDF volumes, we propose a diffusion-based SDF fusion approach. The generated results demonstrate that our work is capable to achieve high-quality room generation directly in three-dimensional space.
• Score: 52.57289166214808
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present DiffInDScene, a novel framework for tackling the problem of high-quality 3D indoor scene generation, which is challenging due to the complexity and diversity of the indoor scene geometry. Although diffusion-based generative models have previously demonstrated impressive performance in image generation and object-level 3D generation, they have not yet been applied to room-level 3D generation due to their computationally intensive costs. In DiffInDScene, we propose a cascaded 3D diffusion pipeline that is efficient and possesses strong generative performance for Truncated Signed Distance Function (TSDF). The whole pipeline is designed to run on a sparse occupancy space in a coarse-to-fine fashion. Inspired by KinectFusion's incremental alignment and fusion of local TSDF volumes, we propose a diffusion-based SDF fusion approach that iteratively diffuses and fuses local TSDF volumes, facilitating the generation of an entire room environment. The generated results demonstrate that our work is capable to achieve high-quality room generation directly in three-dimensional space, starting from scratch. In addition to the scene generation, the final part of DiffInDScene can be used as a post-processing module to refine the 3D reconstruction results from multi-view stereo. According to the user study, the mesh quality generated by our DiffInDScene can even outperform the ground truth mesh provided by ScanNet. Please visit our project page for the latest progress and demonstrations: https://github.com/AkiraHero/diffindscene.

Related papers

• OneWorld: Taming Scene Generation with 3D Unified Representation Autoencoder [90.8453349494245]
  We present OneWorld, a framework that performs diffusion directly within a coherent 3D representation space.<n>OneWorld generates high-quality 3D scenes with superior cross-view consistency compared to state-of-the-art 2D-based methods.
  arXiv  Detail & Related papers  (2026-03-17T03:43:37Z)
• UniLat3D: Geometry-Appearance Unified Latents for Single-Stage 3D Generation [98.40254523605581]
  UniLat3D is a unified framework that encodes geometry and appearance in a single latent space.<n>Our key contribution is a geometry-appearance Unified VAE, which compresses high-resolution sparse features into a compact latent representation.<n>UniLat3D produces high-quality 3D assets in seconds from a single image.
  arXiv  Detail & Related papers  (2025-09-29T17:21:23Z)
• Direct and Explicit 3D Generation from a Single Image [25.207277983430608]
  We introduce a novel framework to directly generate explicit surface geometry and texture using multi-view 2D depth and RGB images. We incorporate epipolar attention into the latent-to-pixel decoder for pixel-level multi-view consistency. By back-projecting the generated depth pixels into 3D space, we create a structured 3D representation.
  arXiv  Detail & Related papers  (2024-11-17T03:14:50Z)
• GaussianAnything: Interactive Point Cloud Latent Diffusion for 3D Generation [75.39457097832113]
  This paper introduces a novel 3D generation framework, offering scalable, high-quality 3D generation with an interactive Point Cloud-structured Latent space. Our framework employs a Variational Autoencoder with multi-view posed RGB-D(epth)-N(ormal) renderings as input, using a unique latent space design that preserves 3D shape information. The proposed method, GaussianAnything, supports multi-modal conditional 3D generation, allowing for point cloud, caption, and single/multi-view image inputs.
  arXiv  Detail & Related papers  (2024-11-12T18:59:32Z)
• LT3SD: Latent Trees for 3D Scene Diffusion [71.91446143124648]
  We present LT3SD, a novel latent diffusion model for large-scale 3D scene generation. We demonstrate the efficacy and benefits of LT3SD for large-scale, high-quality unconditional 3D scene generation.
  arXiv  Detail & Related papers  (2024-09-12T16:55:51Z)
• DiffTF++: 3D-aware Diffusion Transformer for Large-Vocabulary 3D Generation [53.20147419879056]
  We introduce a diffusion-based feed-forward framework to address challenges with a single model. Building upon our 3D-aware Diffusion model with TransFormer, we propose a stronger version for 3D generation, i.e., DiffTF++. Experiments on ShapeNet and OmniObject3D convincingly demonstrate the effectiveness of our proposed modules.
  arXiv  Detail & Related papers  (2024-05-13T17:59:51Z)
• LN3Diff: Scalable Latent Neural Fields Diffusion for Speedy 3D Generation [73.36690511083894]
  This paper introduces a novel framework called LN3Diff to address a unified 3D diffusion pipeline. Our approach harnesses a 3D-aware architecture and variational autoencoder to encode the input image into a structured, compact, and 3D latent space. It achieves state-of-the-art performance on ShapeNet for 3D generation and demonstrates superior performance in monocular 3D reconstruction and conditional 3D generation.
  arXiv  Detail & Related papers  (2024-03-18T17:54:34Z)
• HoloFusion: Towards Photo-realistic 3D Generative Modeling [77.03830223281787]
  Diffusion-based image generators can now produce high-quality and diverse samples, but their success has yet to fully translate to 3D generation. We present HoloFusion, a method that combines the best of these approaches to produce high-fidelity, plausible, and diverse 3D samples.
  arXiv  Detail & Related papers  (2023-08-28T01:19:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.