Text-to-3D using Gaussian Splatting

• URL: http://arxiv.org/abs/2309.16585v4
• Date: Tue, 2 Apr 2024 05:10:02 GMT
• Title: Text-to-3D using Gaussian Splatting
• Authors: Zilong Chen, Feng Wang, Yikai Wang, Huaping Liu,
• Abstract summary: This paper proposes GSGEN, a novel method that adopts Gaussian Splatting, a recent state-of-the-art representation, to text-to-3D generation. GSGEN aims at generating high-quality 3D objects and addressing existing shortcomings by exploiting the explicit nature of Gaussian Splatting. Our approach can generate 3D assets with delicate details and accurate geometry.
• Score: 18.163413810199234
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Automatic text-to-3D generation that combines Score Distillation Sampling (SDS) with the optimization of volume rendering has achieved remarkable progress in synthesizing realistic 3D objects. Yet most existing text-to-3D methods by SDS and volume rendering suffer from inaccurate geometry, e.g., the Janus issue, since it is hard to explicitly integrate 3D priors into implicit 3D representations. Besides, it is usually time-consuming for them to generate elaborate 3D models with rich colors. In response, this paper proposes GSGEN, a novel method that adopts Gaussian Splatting, a recent state-of-the-art representation, to text-to-3D generation. GSGEN aims at generating high-quality 3D objects and addressing existing shortcomings by exploiting the explicit nature of Gaussian Splatting that enables the incorporation of 3D prior. Specifically, our method adopts a progressive optimization strategy, which includes a geometry optimization stage and an appearance refinement stage. In geometry optimization, a coarse representation is established under 3D point cloud diffusion prior along with the ordinary 2D SDS optimization, ensuring a sensible and 3D-consistent rough shape. Subsequently, the obtained Gaussians undergo an iterative appearance refinement to enrich texture details. In this stage, we increase the number of Gaussians by compactness-based densification to enhance continuity and improve fidelity. With these designs, our approach can generate 3D assets with delicate details and accurate geometry. Extensive evaluations demonstrate the effectiveness of our method, especially for capturing high-frequency components. Our code is available at https://github.com/gsgen3d/gsgen

Related papers

• Enhancing Single Image to 3D Generation using Gaussian Splatting and Hybrid Diffusion Priors [17.544733016978928]
  3D object generation from a single image involves estimating the full 3D geometry and texture of unseen views from an unposed RGB image captured in the wild. Recent advancements in 3D object generation have introduced techniques that reconstruct an object's 3D shape and texture. We propose bridging the gap between 2D and 3D diffusion models to address this limitation.
  arXiv  Detail & Related papers  (2024-10-12T10:14:11Z)
• ScalingGaussian: Enhancing 3D Content Creation with Generative Gaussian Splatting [30.99112626706754]
  The creation of high-quality 3D assets is paramount for applications in digital heritage, entertainment, and robotics. Traditionally, this process necessitates skilled professionals and specialized software for modeling. We introduce a novel 3D content creation framework, which generates 3D textures efficiently.
  arXiv  Detail & Related papers  (2024-07-26T18:26:01Z)
• GSD: View-Guided Gaussian Splatting Diffusion for 3D Reconstruction [52.04103235260539]
  We present a diffusion model approach based on Gaussian Splatting representation for 3D object reconstruction from a single view. The model learns to generate 3D objects represented by sets of GS ellipsoids. The final reconstructed objects explicitly come with high-quality 3D structure and texture, and can be efficiently rendered in arbitrary views.
  arXiv  Detail & Related papers  (2024-07-05T03:43:08Z)
• DIRECT-3D: Learning Direct Text-to-3D Generation on Massive Noisy 3D Data [50.164670363633704]
  We present DIRECT-3D, a diffusion-based 3D generative model for creating high-quality 3D assets from text prompts. Our model is directly trained on extensive noisy and unaligned in-the-wild' 3D assets. We achieve state-of-the-art performance in both single-class generation and text-to-3D generation.
  arXiv  Detail & Related papers  (2024-06-06T17:58:15Z)
• GVGEN: Text-to-3D Generation with Volumetric Representation [89.55687129165256]
  3D Gaussian splatting has emerged as a powerful technique for 3D reconstruction and generation, known for its fast and high-quality rendering capabilities. This paper introduces a novel diffusion-based framework, GVGEN, designed to efficiently generate 3D Gaussian representations from text input.
  arXiv  Detail & Related papers  (2024-03-19T17:57:52Z)
• Hyper-3DG: Text-to-3D Gaussian Generation via Hypergraph [20.488040789522604]
  We propose a method named 3D Gaussian Generation via Hypergraph (Hyper-3DG)'', designed to capture the sophisticated high-order correlations present within 3D objects. Our framework allows for the production of finely generated 3D objects within a cohesive optimization, effectively circumventing degradation.
  arXiv  Detail & Related papers  (2024-03-14T09:59:55Z)
• GaussianPro: 3D Gaussian Splatting with Progressive Propagation [49.918797726059545]
  3DGS relies heavily on the point cloud produced by Structure-from-Motion (SfM) techniques. We propose a novel method that applies a progressive propagation strategy to guide the densification of the 3D Gaussians. Our method significantly surpasses 3DGS on the dataset, exhibiting an improvement of 1.15dB in terms of PSNR.
  arXiv  Detail & Related papers  (2024-02-22T16:00:20Z)
• AGG: Amortized Generative 3D Gaussians for Single Image to 3D [108.38567665695027]
  We introduce an Amortized Generative 3D Gaussian framework (AGG) that instantly produces 3D Gaussians from a single image. AGG decomposes the generation of 3D Gaussian locations and other appearance attributes for joint optimization. We propose a cascaded pipeline that first generates a coarse representation of the 3D data and later upsamples it with a 3D Gaussian super-resolution module.
  arXiv  Detail & Related papers  (2024-01-08T18:56:33Z)
• DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creation [55.661467968178066]
  We propose DreamGaussian, a novel 3D content generation framework that achieves both efficiency and quality simultaneously. Our key insight is to design a generative 3D Gaussian Splatting model with companioned mesh extraction and texture refinement in UV space. In contrast to the occupancy pruning used in Neural Radiance Fields, we demonstrate that the progressive densification of 3D Gaussians converges significantly faster for 3D generative tasks.
  arXiv  Detail & Related papers  (2023-09-28T17:55:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.