BloomScene: Lightweight Structured 3D Gaussian Splatting for Crossmodal Scene Generation

• URL: http://arxiv.org/abs/2501.10462v1
• Date: Wed, 15 Jan 2025 11:33:34 GMT
• Title: BloomScene: Lightweight Structured 3D Gaussian Splatting for Crossmodal Scene Generation
• Authors: Xiaolu Hou, Mingcheng Li, Dingkang Yang, Jiawei Chen, Ziyun Qian, Xiao Zhao, Yue Jiang, Jinjie Wei, Qingyao Xu, Lihua Zhang,
• Abstract summary: 3D scenes have highly complex structures and need to ensure that the output is dense, coherent, and contains all necessary structures. Current 3D scene generation methods rely on pre-trained text-to-image diffusion models and monocular depth estimators. We propose BloomScene, a lightweight structured 3D Gaussian splatting for crossmodal scene generation.
• Score: 16.00575923179227
• License:
• Abstract: With the widespread use of virtual reality applications, 3D scene generation has become a new challenging research frontier. 3D scenes have highly complex structures and need to ensure that the output is dense, coherent, and contains all necessary structures. Many current 3D scene generation methods rely on pre-trained text-to-image diffusion models and monocular depth estimators. However, the generated scenes occupy large amounts of storage space and often lack effective regularisation methods, leading to geometric distortions. To this end, we propose BloomScene, a lightweight structured 3D Gaussian splatting for crossmodal scene generation, which creates diverse and high-quality 3D scenes from text or image inputs. Specifically, a crossmodal progressive scene generation framework is proposed to generate coherent scenes utilizing incremental point cloud reconstruction and 3D Gaussian splatting. Additionally, we propose a hierarchical depth prior-based regularization mechanism that utilizes multi-level constraints on depth accuracy and smoothness to enhance the realism and continuity of the generated scenes. Ultimately, we propose a structured context-guided compression mechanism that exploits structured hash grids to model the context of unorganized anchor attributes, which significantly eliminates structural redundancy and reduces storage overhead. Comprehensive experiments across multiple scenes demonstrate the significant potential and advantages of our framework compared with several baselines.

Related papers

• VideoLifter: Lifting Videos to 3D with Fast Hierarchical Stereo Alignment [62.6737516863285]
  VideoLifter is a novel framework that incrementally optimize a globally sparse to dense 3D representation directly from video sequences. By tracking and propagating sparse point correspondences across frames and fragments, VideoLifter incrementally refines camera poses and 3D structure. This approach significantly accelerates the reconstruction process, reducing training time by over 82% while surpassing current state-of-the-art methods in visual fidelity and computational efficiency.
  arXiv  Detail & Related papers  (2025-01-03T18:52:36Z)
• CATSplat: Context-Aware Transformer with Spatial Guidance for Generalizable 3D Gaussian Splatting from A Single-View Image [18.445769892372528]
  We introduce CATSplat, a novel generalizable transformer-based framework for single-view 3D scene reconstruction. By incorporating scene-specific contextual details from text embeddings through cross-attention, we pave the way for context-aware reconstruction. Experiments on large-scale datasets demonstrate the state-of-the-art performance of CATSplat in single-view 3D scene reconstruction.
  arXiv  Detail & Related papers  (2024-12-17T13:32:04Z)
• Wonderland: Navigating 3D Scenes from a Single Image [43.99037613068823]
  We introduce a large-scale reconstruction model that uses latents from a video diffusion model to predict 3D Gaussian Splattings for the scenes. We train the 3D reconstruction model to operate on the video latent space with a progressive training strategy, enabling the efficient generation of high-quality, wide-scope, and generic 3D scenes.
  arXiv  Detail & Related papers  (2024-12-16T18:58:17Z)
• SplatFields: Neural Gaussian Splats for Sparse 3D and 4D Reconstruction [24.33543853742041]
  3D Gaussian Splatting (3DGS) has emerged as a practical and scalable reconstruction method. We propose an optimization strategy that effectively regularizes splat features by modeling them as the outputs of a corresponding implicit neural field. Our approach effectively handles static and dynamic cases, as demonstrated by extensive testing across different setups and scene complexities.
  arXiv  Detail & Related papers  (2024-09-17T14:04:20Z)
• LT3SD: Latent Trees for 3D Scene Diffusion [71.91446143124648]
  We present LT3SD, a novel latent diffusion model for large-scale 3D scene generation. We demonstrate the efficacy and benefits of LT3SD for large-scale, high-quality unconditional 3D scene generation.
  arXiv  Detail & Related papers  (2024-09-12T16:55:51Z)
• ViewCrafter: Taming Video Diffusion Models for High-fidelity Novel View Synthesis [63.169364481672915]
  We propose textbfViewCrafter, a novel method for synthesizing high-fidelity novel views of generic scenes from single or sparse images. Our method takes advantage of the powerful generation capabilities of video diffusion model and the coarse 3D clues offered by point-based representation to generate high-quality video frames.
  arXiv  Detail & Related papers  (2024-09-03T16:53:19Z)
• HoloDreamer: Holistic 3D Panoramic World Generation from Text Descriptions [31.342899807980654]
  3D scene generation is in high demand across various domains, including virtual reality, gaming, and the film industry. We introduce HoloDreamer, a framework that first generates high-definition panorama as a holistic initialization of the full 3D scene. We then leverage 3D Gaussian Splatting (3D-GS) to quickly reconstruct the 3D scene, thereby facilitating the creation of view-consistent and fully enclosed 3D scenes.
  arXiv  Detail & Related papers  (2024-07-21T14:52:51Z)
• 3D-SceneDreamer: Text-Driven 3D-Consistent Scene Generation [51.64796781728106]
  We propose a generative refinement network to synthesize new contents with higher quality by exploiting the natural image prior to 2D diffusion model and the global 3D information of the current scene. Our approach supports wide variety of scene generation and arbitrary camera trajectories with improved visual quality and 3D consistency.
  arXiv  Detail & Related papers  (2024-03-14T14:31:22Z)
• Denoising Diffusion via Image-Based Rendering [54.20828696348574]
  We introduce the first diffusion model able to perform fast, detailed reconstruction and generation of real-world 3D scenes. First, we introduce a new neural scene representation, IB-planes, that can efficiently and accurately represent large 3D scenes. Second, we propose a denoising-diffusion framework to learn a prior over this novel 3D scene representation, using only 2D images.
  arXiv  Detail & Related papers  (2024-02-05T19:00:45Z)
• SceneWiz3D: Towards Text-guided 3D Scene Composition [134.71933134180782]
  Existing approaches either leverage large text-to-image models to optimize a 3D representation or train 3D generators on object-centric datasets. We introduce SceneWiz3D, a novel approach to synthesize high-fidelity 3D scenes from text.
  arXiv  Detail & Related papers  (2023-12-13T18:59:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.