Text2Immersion: Generative Immersive Scene with 3D Gaussians

• URL: http://arxiv.org/abs/2312.09242v1
• Date: Thu, 14 Dec 2023 18:58:47 GMT
• Title: Text2Immersion: Generative Immersive Scene with 3D Gaussians
• Authors: Hao Ouyang, Kathryn Heal, Stephen Lombardi, Tiancheng Sun
• Abstract summary: Text2Immersion is an elegant method for producing high-quality 3D immersive scenes from text prompts. Our system surpasses other methods in rendering quality and diversity, further progressing towards text-driven 3D scene generation.
• Score: 14.014016090679627
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce Text2Immersion, an elegant method for producing high-quality 3D immersive scenes from text prompts. Our proposed pipeline initiates by progressively generating a Gaussian cloud using pre-trained 2D diffusion and depth estimation models. This is followed by a refining stage on the Gaussian cloud, interpolating and refining it to enhance the details of the generated scene. Distinct from prevalent methods that focus on single object or indoor scenes, or employ zoom-out trajectories, our approach generates diverse scenes with various objects, even extending to the creation of imaginary scenes. Consequently, Text2Immersion can have wide-ranging implications for various applications such as virtual reality, game development, and automated content creation. Extensive evaluations demonstrate that our system surpasses other methods in rendering quality and diversity, further progressing towards text-driven 3D scene generation. We will make the source code publicly accessible at the project page.

Related papers

• InsTex: Indoor Scenes Stylized Texture Synthesis [81.12010726769768]
  High-quality textures are crucial for 3D scenes for augmented/virtual reality (ARVR) applications. Current methods suffer from lengthy processing times and visual artifacts. We introduce two-stage architecture designed to generate high-quality textures for 3D scenes.
  arXiv  Detail & Related papers  (2025-01-22T08:37:59Z)
• Layout2Scene: 3D Semantic Layout Guided Scene Generation via Geometry and Appearance Diffusion Priors [52.63385546943866]
  We present a text-to-scene generation method (namely, Layout2Scene) using additional semantic layout as the prompt to inject precise control of 3D object positions. To fully leverage 2D diffusion priors in geometry and appearance generation, we introduce a semantic-guided geometry diffusion model and a semantic-geometry guided diffusion model. Our method can generate more plausible and realistic scenes as compared to state-of-the-art approaches.
  arXiv  Detail & Related papers  (2025-01-05T12:20:13Z)
• TexAVi: Generating Stereoscopic VR Video Clips from Text Descriptions [0.562479170374811]
  This paper proposes an approach to coalesce existing generative systems to form a stereoscopic virtual reality video from text. Our work highlights the exciting possibilities of using natural language-driven graphics in fields like virtual reality simulations.
  arXiv  Detail & Related papers  (2025-01-02T09:21:03Z)
• Architect: Generating Vivid and Interactive 3D Scenes with Hierarchical 2D Inpainting [47.014044892025346]
  Architect is a generative framework that creates complex and realistic 3D embodied environments leveraging diffusion-based 2D image inpainting. Our pipeline is further extended to a hierarchical and iterative inpainting process to continuously generate placement of large furniture and small objects to enrich the scene.
  arXiv  Detail & Related papers  (2024-11-14T22:15:48Z)
• SceneDreamer360: Text-Driven 3D-Consistent Scene Generation with Panoramic Gaussian Splatting [53.32467009064287]
  We propose a text-driven 3D-consistent scene generation model: SceneDreamer360. Our proposed method leverages a text-driven panoramic image generation model as a prior for 3D scene generation. Our experiments demonstrate that SceneDreamer360 with its panoramic image generation and 3DGS can produce higher quality, spatially consistent, and visually appealing 3D scenes from any text prompt.
  arXiv  Detail & Related papers  (2024-08-25T02:56:26Z)
• Sketch2Scene: Automatic Generation of Interactive 3D Game Scenes from User's Casual Sketches [50.51643519253066]
  3D Content Generation is at the heart of many computer graphics applications, including video gaming, film-making, virtual and augmented reality, etc. This paper proposes a novel deep-learning based approach for automatically generating interactive and playable 3D game scenes.
  arXiv  Detail & Related papers  (2024-08-08T16:27:37Z)
• HoloDreamer: Holistic 3D Panoramic World Generation from Text Descriptions [31.342899807980654]
  3D scene generation is in high demand across various domains, including virtual reality, gaming, and the film industry. We introduce HoloDreamer, a framework that first generates high-definition panorama as a holistic initialization of the full 3D scene. We then leverage 3D Gaussian Splatting (3D-GS) to quickly reconstruct the 3D scene, thereby facilitating the creation of view-consistent and fully enclosed 3D scenes.
  arXiv  Detail & Related papers  (2024-07-21T14:52:51Z)
• DreamScape: 3D Scene Creation via Gaussian Splatting joint Correlation Modeling [23.06464506261766]
  We present DreamScape, a method for creating highly consistent 3D scenes solely from textual descriptions. Our approach involves a 3D Gaussian Guide for scene representation, consisting of semantic primitives (objects) and their spatial transformations. A progressive scale control is tailored during local object generation, ensuring that objects of different sizes and densities adapt to the scene.
  arXiv  Detail & Related papers  (2024-04-14T12:13:07Z)
• 3D-SceneDreamer: Text-Driven 3D-Consistent Scene Generation [51.64796781728106]
  We propose a generative refinement network to synthesize new contents with higher quality by exploiting the natural image prior to 2D diffusion model and the global 3D information of the current scene. Our approach supports wide variety of scene generation and arbitrary camera trajectories with improved visual quality and 3D consistency.
  arXiv  Detail & Related papers  (2024-03-14T14:31:22Z)
• ViewDiff: 3D-Consistent Image Generation with Text-to-Image Models [65.22994156658918]
  We present a method that learns to generate multi-view images in a single denoising process from real-world data. We design an autoregressive generation that renders more 3D-consistent images at any viewpoint.
  arXiv  Detail & Related papers  (2024-03-04T07:57:05Z)
• SceneWiz3D: Towards Text-guided 3D Scene Composition [134.71933134180782]
  Existing approaches either leverage large text-to-image models to optimize a 3D representation or train 3D generators on object-centric datasets. We introduce SceneWiz3D, a novel approach to synthesize high-fidelity 3D scenes from text.
  arXiv  Detail & Related papers  (2023-12-13T18:59:30Z)
• LucidDreamer: Domain-free Generation of 3D Gaussian Splatting Scenes [52.31402192831474]
  Existing 3D scene generation models, however, limit the target scene to specific domain. We propose LucidDreamer, a domain-free scene generation pipeline. LucidDreamer produces highly-detailed Gaussian splats with no constraint on domain of the target scene.
  arXiv  Detail & Related papers  (2023-11-22T13:27:34Z)
• Static and Animated 3D Scene Generation from Free-form Text Descriptions [1.102914654802229]
  We study a new pipeline that aims to generate static as well as animated 3D scenes from different types of free-form textual scene description. In the first stage, we encode the free-form text using an encoder-decoder neural architecture. In the second stage, we generate a 3D scene based on the generated encoding.
  arXiv  Detail & Related papers  (2020-10-04T11:31:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.