DiffGS: Functional Gaussian Splatting Diffusion

• URL: http://arxiv.org/abs/2410.19657v2
• Date: Wed, 30 Oct 2024 03:34:49 GMT
• Title: DiffGS: Functional Gaussian Splatting Diffusion
• Authors: Junsheng Zhou, Weiqi Zhang, Yu-Shen Liu,
• Abstract summary: 3D Gaussian Splatting (3DGS) has shown convincing performance in rendering speed and fidelity. However, the generation of Gaussian Splatting remains a challenge due to its discreteness and unstructured nature. We propose DiffGS, a general Gaussian generator based on latent diffusion models.
• Score: 33.07847512591061
• License:
• Abstract: 3D Gaussian Splatting (3DGS) has shown convincing performance in rendering speed and fidelity, yet the generation of Gaussian Splatting remains a challenge due to its discreteness and unstructured nature. In this work, we propose DiffGS, a general Gaussian generator based on latent diffusion models. DiffGS is a powerful and efficient 3D generative model which is capable of generating Gaussian primitives at arbitrary numbers for high-fidelity rendering with rasterization. The key insight is to represent Gaussian Splatting in a disentangled manner via three novel functions to model Gaussian probabilities, colors and transforms. Through the novel disentanglement of 3DGS, we represent the discrete and unstructured 3DGS with continuous Gaussian Splatting functions, where we then train a latent diffusion model with the target of generating these Gaussian Splatting functions both unconditionally and conditionally. Meanwhile, we introduce a discretization algorithm to extract Gaussians at arbitrary numbers from the generated functions via octree-guided sampling and optimization. We explore DiffGS for various tasks, including unconditional generation, conditional generation from text, image, and partial 3DGS, as well as Point-to-Gaussian generation. We believe that DiffGS provides a new direction for flexibly modeling and generating Gaussian Splatting.

Related papers

• GaussianSpa: An "Optimizing-Sparsifying" Simplification Framework for Compact and High-Quality 3D Gaussian Splatting [12.342660713851227]
  3D Gaussian Splatting (3DGS) has emerged as a mainstream for novel view synthesis, leveraging continuous aggregations of Gaussian functions. 3DGS suffers from substantial memory requirements to store the multitude of Gaussians, hindering its practicality. We introduce GaussianSpa, an optimization-based simplification framework for compact and high-quality 3DGS.
  arXiv  Detail & Related papers  (2024-11-09T00:38:06Z)
• PixelGaussian: Generalizable 3D Gaussian Reconstruction from Arbitrary Views [116.10577967146762]
  PixelGaussian is an efficient framework for learning generalizable 3D Gaussian reconstruction from arbitrary views. Our method achieves state-of-the-art performance with good generalization to various numbers of views.
  arXiv  Detail & Related papers  (2024-10-24T17:59:58Z)
• L3DG: Latent 3D Gaussian Diffusion [74.36431175937285]
  L3DG is the first approach for generative 3D modeling of 3D Gaussians through a latent 3D Gaussian diffusion formulation. We employ a sparse convolutional architecture to efficiently operate on room-scale scenes. By leveraging the 3D Gaussian representation, the generated scenes can be rendered from arbitrary viewpoints in real-time.
  arXiv  Detail & Related papers  (2024-10-17T13:19:32Z)
• Variational Bayes Gaussian Splatting [44.43761190929142]
  3D Gaussian Splatting has emerged as a promising approach for modeling 3D scenes using mixtures of Gaussians. We propose Variational Bayes Gaussian Splatting, a novel approach that frames training a Gaussian splat as variational inference over model parameters. Our experiments show that VBGS not only matches state-of-the-art performance on static datasets, but also enables continual learning from sequentially streamed 2D and 3D data.
  arXiv  Detail & Related papers  (2024-10-04T16:52:03Z)
• GaussianForest: Hierarchical-Hybrid 3D Gaussian Splatting for Compressed Scene Modeling [40.743135560583816]
  We introduce the Gaussian-Forest modeling framework, which hierarchically represents a scene as a forest of hybrid 3D Gaussians. Experiments demonstrate that Gaussian-Forest not only maintains comparable speed and quality but also achieves a compression rate surpassing 10 times.
  arXiv  Detail & Related papers  (2024-06-13T02:41:11Z)
• Gaussian Opacity Fields: Efficient Adaptive Surface Reconstruction in Unbounded Scenes [50.92217884840301]
  Gaussian Opacity Fields (GOF) is a novel approach for efficient, high-quality, and adaptive surface reconstruction in scenes. GOF is derived from ray-tracing-based volume rendering of 3D Gaussians. GOF surpasses existing 3DGS-based methods in surface reconstruction and novel view synthesis.
  arXiv  Detail & Related papers  (2024-04-16T17:57:19Z)
• GaussianCube: A Structured and Explicit Radiance Representation for 3D Generative Modeling [55.05713977022407]
  We introduce a radiance representation that is both structured and fully explicit and thus greatly facilitates 3D generative modeling. We derive GaussianCube by first using a novel densification-constrained Gaussian fitting algorithm, which yields high-accuracy fitting. Experiments conducted on unconditional and class-conditioned object generation, digital avatar creation, and text-to-3D all show that our model synthesis achieves state-of-the-art generation results.
  arXiv  Detail & Related papers  (2024-03-28T17:59:50Z)
• GVGEN: Text-to-3D Generation with Volumetric Representation [89.55687129165256]
  3D Gaussian splatting has emerged as a powerful technique for 3D reconstruction and generation, known for its fast and high-quality rendering capabilities. This paper introduces a novel diffusion-based framework, GVGEN, designed to efficiently generate 3D Gaussian representations from text input.
  arXiv  Detail & Related papers  (2024-03-19T17:57:52Z)
• GES: Generalized Exponential Splatting for Efficient Radiance Field Rendering [112.16239342037714]
  GES (Generalized Exponential Splatting) is a novel representation that employs Generalized Exponential Function (GEF) to model 3D scenes. With the aid of a frequency-modulated loss, GES achieves competitive performance in novel-view synthesis benchmarks.
  arXiv  Detail & Related papers  (2024-02-15T17:32:50Z)
• GaussianDiffusion: 3D Gaussian Splatting for Denoising Diffusion Probabilistic Models with Structured Noise [0.0]
  This paper introduces a novel text to 3D content generation framework based on Gaussian splatting. The challenge of achieving multi-view consistency in 3D generation significantly impedes modeling complexity and accuracy.
  arXiv  Detail & Related papers  (2023-11-19T04:26:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.