DreamScape: 3D Scene Creation via Gaussian Splatting joint Correlation Modeling

• URL: http://arxiv.org/abs/2404.09227v3
• Date: Wed, 02 Apr 2025 14:54:24 GMT
• Title: DreamScape: 3D Scene Creation via Gaussian Splatting joint Correlation Modeling
• Authors: Yueming Zhao, Xuening Yuan, Hongyu Yang, Di Huang,
• Abstract summary: We present DreamScape, a method for generating 3D scenes from text.<n>We use 3D Gaussian Guide that encodes semantic primitives, spatial transformations and relationships from text using LLMs.<n>DreamScape achieves state-of-the-art performance, enabling high-fidelity, controllable 3D scene generation.
• Score: 23.06464506261766
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent advances in text-to-3D creation integrate the potent prior of Diffusion Models from text-to-image generation into 3D domain. Nevertheless, generating 3D scenes with multiple objects remains challenging. Therefore, we present DreamScape, a method for generating 3D scenes from text. Utilizing Gaussian Splatting for 3D representation, DreamScape introduces 3D Gaussian Guide that encodes semantic primitives, spatial transformations and relationships from text using LLMs, enabling local-to-global optimization. Progressive scale control is tailored during local object generation, addressing training instability issue arising from simple blending in the global optimization stage. Collision relationships between objects are modeled at the global level to mitigate biases in LLMs priors, ensuring physical correctness. Additionally, to generate pervasive objects like rain and snow distributed extensively across the scene, we design specialized sparse initialization and densification strategy. Experiments demonstrate that DreamScape achieves state-of-the-art performance, enabling high-fidelity, controllable 3D scene generation.

Related papers

• Can3Tok: Canonical 3D Tokenization and Latent Modeling of Scene-Level 3D Gaussians [27.19772539224761]
  We introduce Can3Tok, the first 3D scene-level variational autoencoder capable of encoding a large number of Gaussian primitives into a low-dimensional latent embedding.<n>We propose a general pipeline for 3D scene data processing to address scale inconsistency issue.
  arXiv  Detail & Related papers  (2025-08-02T18:43:45Z)
• GSD: View-Guided Gaussian Splatting Diffusion for 3D Reconstruction [52.04103235260539]
  We present a diffusion model approach based on Gaussian Splatting representation for 3D object reconstruction from a single view. The model learns to generate 3D objects represented by sets of GS ellipsoids. The final reconstructed objects explicitly come with high-quality 3D structure and texture, and can be efficiently rendered in arbitrary views.
  arXiv  Detail & Related papers  (2024-07-05T03:43:08Z)
• Enhancing Generalizability of Representation Learning for Data-Efficient 3D Scene Understanding [50.448520056844885]
  We propose a generative Bayesian network to produce diverse synthetic scenes with real-world patterns. A series of experiments robustly display our method's consistent superiority over existing state-of-the-art pre-training approaches.
  arXiv  Detail & Related papers  (2024-06-17T07:43:53Z)
• RealmDreamer: Text-Driven 3D Scene Generation with Inpainting and Depth Diffusion [39.03289977892935]
  RealmDreamer is a technique for generation of general forward-facing 3D scenes from text descriptions. Our technique does not require video or multi-view data and can synthesize a variety of high-quality 3D scenes in different styles.
  arXiv  Detail & Related papers  (2024-04-10T17:57:41Z)
• Planner3D: LLM-enhanced graph prior meets 3D indoor scene explicit regularization [31.52569918586902]
  3D scene synthesis has diverse applications across a spectrum of industries such as robotics, films, and video games. In this paper, we aim at generating realistic and reasonable 3D indoor scenes from scene graph. Our method achieves better 3D scene synthesis, especially in terms of scene-level fidelity.
  arXiv  Detail & Related papers  (2024-03-19T15:54:48Z)
• GALA3D: Towards Text-to-3D Complex Scene Generation via Layout-guided Generative Gaussian Splatting [52.150502668874495]
  We present GALA3D, generative 3D GAussians with LAyout-guided control, for effective compositional text-to-3D generation. GALA3D is a user-friendly, end-to-end framework for state-of-the-art scene-level 3D content generation and controllable editing.
  arXiv  Detail & Related papers  (2024-02-11T13:40:08Z)
• GS-CLIP: Gaussian Splatting for Contrastive Language-Image-3D Pretraining from Real-World Data [73.06536202251915]
  3D Shape represented as point cloud has achieve advancements in multimodal pre-training to align image and language descriptions. We propose GS-CLIP for the first attempt to introduce 3DGS into multimodal pre-training to enhance 3D representation.
  arXiv  Detail & Related papers  (2024-02-09T05:46:47Z)
• Denoising Diffusion via Image-Based Rendering [54.20828696348574]
  We introduce the first diffusion model able to perform fast, detailed reconstruction and generation of real-world 3D scenes. First, we introduce a new neural scene representation, IB-planes, that can efficiently and accurately represent large 3D scenes. Second, we propose a denoising-diffusion framework to learn a prior over this novel 3D scene representation, using only 2D images.
  arXiv  Detail & Related papers  (2024-02-05T19:00:45Z)
• FMGS: Foundation Model Embedded 3D Gaussian Splatting for Holistic 3D Scene Understanding [11.118857208538039]
  We present Foundation Model Embedded Gaussian Splatting (S), which incorporates vision-language embeddings of foundation models into 3D Gaussian Splatting (GS) Results demonstrate remarkable multi-view semantic consistency, facilitating diverse downstream tasks, beating state-of-the-art methods by 10.2 percent on open-vocabulary language-based object detection. This research explores the intersection of vision, language, and 3D scene representation, paving the way for enhanced scene understanding in uncontrolled real-world environments.
  arXiv  Detail & Related papers  (2024-01-03T20:39:02Z)
• SceneWiz3D: Towards Text-guided 3D Scene Composition [134.71933134180782]
  Existing approaches either leverage large text-to-image models to optimize a 3D representation or train 3D generators on object-centric datasets. We introduce SceneWiz3D, a novel approach to synthesize high-fidelity 3D scenes from text.
  arXiv  Detail & Related papers  (2023-12-13T18:59:30Z)
• LucidDreamer: Domain-free Generation of 3D Gaussian Splatting Scenes [52.31402192831474]
  Existing 3D scene generation models, however, limit the target scene to specific domain. We propose LucidDreamer, a domain-free scene generation pipeline. LucidDreamer produces highly-detailed Gaussian splats with no constraint on domain of the target scene.
  arXiv  Detail & Related papers  (2023-11-22T13:27:34Z)
• 3DStyle-Diffusion: Pursuing Fine-grained Text-driven 3D Stylization with 2D Diffusion Models [102.75875255071246]
  3D content creation via text-driven stylization has played a fundamental challenge to multimedia and graphics community. We propose a new 3DStyle-Diffusion model that triggers fine-grained stylization of 3D meshes with additional controllable appearance and geometric guidance from 2D Diffusion models.
  arXiv  Detail & Related papers  (2023-11-09T15:51:27Z)
• Text-to-3D using Gaussian Splatting [18.163413810199234]
  This paper proposes GSGEN, a novel method that adopts Gaussian Splatting, a recent state-of-the-art representation, to text-to-3D generation. GSGEN aims at generating high-quality 3D objects and addressing existing shortcomings by exploiting the explicit nature of Gaussian Splatting. Our approach can generate 3D assets with delicate details and accurate geometry.
  arXiv  Detail & Related papers  (2023-09-28T16:44:31Z)
• Differentiable Blocks World: Qualitative 3D Decomposition by Rendering Primitives [70.32817882783608]
  We present an approach that produces a simple, compact, and actionable 3D world representation by means of 3D primitives. Unlike existing primitive decomposition methods that rely on 3D input data, our approach operates directly on images. We show that the resulting textured primitives faithfully reconstruct the input images and accurately model the visible 3D points.
  arXiv  Detail & Related papers  (2023-07-11T17:58:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.