DreamScene: 3D Gaussian-based Text-to-3D Scene Generation via Formation Pattern Sampling

• URL: http://arxiv.org/abs/2404.03575v2
• Date: Fri, 19 Jul 2024 07:28:03 GMT
• Title: DreamScene: 3D Gaussian-based Text-to-3D Scene Generation via Formation Pattern Sampling
• Authors: Haoran Li, Haolin Shi, Wenli Zhang, Wenjun Wu, Yong Liao, Lin Wang, Lik-hang Lee, Pengyuan Zhou,
• Abstract summary: We propose a novel text-to-3D scene generation framework, DreamScene, to tackle the aforementioned three challenges mainly via two strategies. First, DreamScene employs Formation Pattern Sampling (FPS), a multi-timestep sampling strategy guided by the formation patterns of 3D objects, to form fast, semantically rich, and high-quality representations. Second, DreamScene employs a progressive three-stage camera sampling strategy, specifically designed for both indoor and outdoor settings, to effectively ensure object-environment integration and scene-wide 3D consistency.
• Score: 17.807481666320825
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Text-to-3D scene generation holds immense potential for the gaming, film, and architecture sectors. Despite significant progress, existing methods struggle with maintaining high quality, consistency, and editing flexibility. In this paper, we propose DreamScene, a 3D Gaussian-based novel text-to-3D scene generation framework, to tackle the aforementioned three challenges mainly via two strategies. First, DreamScene employs Formation Pattern Sampling (FPS), a multi-timestep sampling strategy guided by the formation patterns of 3D objects, to form fast, semantically rich, and high-quality representations. FPS uses 3D Gaussian filtering for optimization stability, and leverages reconstruction techniques to generate plausible textures. Second, DreamScene employs a progressive three-stage camera sampling strategy, specifically designed for both indoor and outdoor settings, to effectively ensure object-environment integration and scene-wide 3D consistency. Last, DreamScene enhances scene editing flexibility by integrating objects and environments, enabling targeted adjustments. Extensive experiments validate DreamScene's superiority over current state-of-the-art techniques, heralding its wide-ranging potential for diverse applications. Code and demos will be released at https://dreamscene-project.github.io .

Related papers

• DreamScene: 3D Gaussian-based End-to-end Text-to-3D Scene Generation [19.817968922757007]
  We present DreamScene, an end-to-end framework for high-quality and editable 3D scene generation from text or dialogue.<n>To ensure global consistent, DreamScene employs a progressive camera sampling strategy tailored to both indoor and outdoor settings.<n>Experiments demonstrate that DreamScene surpasses prior methods in quality, consistency, and flexibility.
  arXiv  Detail & Related papers  (2025-07-18T14:45:54Z)
• DreamAnywhere: Object-Centric Panoramic 3D Scene Generation [7.118837433208326]
  We present DreamAnywhere, a modular system for the fast generation and prototyping of 3D scenes.<n>Our system supports immersive navigation and intuitive object-level editing, making it ideal for scene exploration, visual mock-ups, and rapid prototyping.<n>Compared to current state-of-the-art text and image-based 3D scene generation approaches, DreamAnywhere shows significant improvements in coherence in novel view synthesis and achieves competitive image quality.
  arXiv  Detail & Related papers  (2025-06-25T12:30:41Z)
• ArtiScene: Language-Driven Artistic 3D Scene Generation Through Image Intermediary [37.41274496314127]
  ArtiScene is a training-free automated pipeline for scene design.<n>It generates 2D images from a scene description, then extract the shape and appearance of objects to create 3D models.<n>It outperforms state-of-the-art benchmarks by a large margin in layout and aesthetic quality by quantitative metrics.
  arXiv  Detail & Related papers  (2025-05-31T23:03:54Z)
• SceneDreamer360: Text-Driven 3D-Consistent Scene Generation with Panoramic Gaussian Splatting [53.32467009064287]
  We propose a text-driven 3D-consistent scene generation model: SceneDreamer360. Our proposed method leverages a text-driven panoramic image generation model as a prior for 3D scene generation. Our experiments demonstrate that SceneDreamer360 with its panoramic image generation and 3DGS can produce higher quality, spatially consistent, and visually appealing 3D scenes from any text prompt.
  arXiv  Detail & Related papers  (2024-08-25T02:56:26Z)
• 3DitScene: Editing Any Scene via Language-guided Disentangled Gaussian Splatting [100.94916668527544]
  Existing methods solely focus on either 2D individual object or 3D global scene editing. We propose 3DitScene, a novel and unified scene editing framework. It enables seamless editing from 2D to 3D, allowing precise control over scene composition and individual objects.
  arXiv  Detail & Related papers  (2024-05-28T17:59:01Z)
• 3D-SceneDreamer: Text-Driven 3D-Consistent Scene Generation [51.64796781728106]
  We propose a generative refinement network to synthesize new contents with higher quality by exploiting the natural image prior to 2D diffusion model and the global 3D information of the current scene. Our approach supports wide variety of scene generation and arbitrary camera trajectories with improved visual quality and 3D consistency.
  arXiv  Detail & Related papers  (2024-03-14T14:31:22Z)
• SceneWiz3D: Towards Text-guided 3D Scene Composition [134.71933134180782]
  Existing approaches either leverage large text-to-image models to optimize a 3D representation or train 3D generators on object-centric datasets. We introduce SceneWiz3D, a novel approach to synthesize high-fidelity 3D scenes from text.
  arXiv  Detail & Related papers  (2023-12-13T18:59:30Z)
• LucidDreamer: Domain-free Generation of 3D Gaussian Splatting Scenes [52.31402192831474]
  Existing 3D scene generation models, however, limit the target scene to specific domain. We propose LucidDreamer, a domain-free scene generation pipeline. LucidDreamer produces highly-detailed Gaussian splats with no constraint on domain of the target scene.
  arXiv  Detail & Related papers  (2023-11-22T13:27:34Z)
• DreamCraft3D: Hierarchical 3D Generation with Bootstrapped Diffusion Prior [40.67100127167502]
  We present DreamCraft3D, a hierarchical 3D content generation method that produces high-fidelity and coherent 3D objects. We tackle the problem by leveraging a 2D reference image to guide the stages of geometry sculpting and texture boosting. With tailored 3D priors throughout the hierarchical generation, DreamCraft3D generates coherent 3D objects with photorealistic renderings.
  arXiv  Detail & Related papers  (2023-10-25T17:50:10Z)
• SceneDreamer: Unbounded 3D Scene Generation from 2D Image Collections [49.802462165826554]
  We present SceneDreamer, an unconditional generative model for unbounded 3D scenes. Our framework is learned from in-the-wild 2D image collections only, without any 3D annotations.
  arXiv  Detail & Related papers  (2023-02-02T18:59:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.