AGG: Amortized Generative 3D Gaussians for Single Image to 3D

• URL: http://arxiv.org/abs/2401.04099v1
• Date: Mon, 8 Jan 2024 18:56:33 GMT
• Title: AGG: Amortized Generative 3D Gaussians for Single Image to 3D
• Authors: Dejia Xu, Ye Yuan, Morteza Mardani, Sifei Liu, Jiaming Song, Zhangyang Wang, Arash Vahdat
• Abstract summary: We introduce an Amortized Generative 3D Gaussian framework (AGG) that instantly produces 3D Gaussians from a single image. AGG decomposes the generation of 3D Gaussian locations and other appearance attributes for joint optimization. We propose a cascaded pipeline that first generates a coarse representation of the 3D data and later upsamples it with a 3D Gaussian super-resolution module.
• Score: 108.38567665695027
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Given the growing need for automatic 3D content creation pipelines, various 3D representations have been studied to generate 3D objects from a single image. Due to its superior rendering efficiency, 3D Gaussian splatting-based models have recently excelled in both 3D reconstruction and generation. 3D Gaussian splatting approaches for image to 3D generation are often optimization-based, requiring many computationally expensive score-distillation steps. To overcome these challenges, we introduce an Amortized Generative 3D Gaussian framework (AGG) that instantly produces 3D Gaussians from a single image, eliminating the need for per-instance optimization. Utilizing an intermediate hybrid representation, AGG decomposes the generation of 3D Gaussian locations and other appearance attributes for joint optimization. Moreover, we propose a cascaded pipeline that first generates a coarse representation of the 3D data and later upsamples it with a 3D Gaussian super-resolution module. Our method is evaluated against existing optimization-based 3D Gaussian frameworks and sampling-based pipelines utilizing other 3D representations, where AGG showcases competitive generation abilities both qualitatively and quantitatively while being several orders of magnitude faster. Project page: https://ir1d.github.io/AGG/

Related papers

• 3D-GSW: 3D Gaussian Splatting Watermark for Protecting Copyrights in Radiance Fields [5.52538716292462]
  We present a novel watermarking method for 3D Gaussian splatting. The proposed method embeds a binary message into 3D Gaussians by fine-tuning the pre-trained 3D Gaussian splatting model. Our experiments show that our method can embed a watermark in 3D Gaussians imperceptibly with increased capacity and robustness against attacks.
  arXiv  Detail & Related papers  (2024-09-20T05:16:06Z)
• GSD: View-Guided Gaussian Splatting Diffusion for 3D Reconstruction [52.04103235260539]
  We present a diffusion model approach based on Gaussian Splatting representation for 3D object reconstruction from a single view. The model learns to generate 3D objects represented by sets of GS ellipsoids. The final reconstructed objects explicitly come with high-quality 3D structure and texture, and can be efficiently rendered in arbitrary views.
  arXiv  Detail & Related papers  (2024-07-05T03:43:08Z)
• GGHead: Fast and Generalizable 3D Gaussian Heads [48.967905053963385]
  3D GANs struggle to scale to generate samples at high resolutions due to their relatively slow train and render speeds. We propose Generative Gaussian Heads (GGHead), which adopts the recent 3D Gaussian Splatting representation within a 3D GAN framework. We demonstrate real-time generation and rendering of high-quality 3D-consistent heads at $10242$ resolution for the first time.
  arXiv  Detail & Related papers  (2024-06-13T17:54:38Z)
• Adversarial Generation of Hierarchical Gaussians for 3D Generative Model [20.833116566243408]
  In this paper, we exploit Gaussian as a 3D representation for 3D GANs by leveraging its efficient and explicit characteristics. We introduce a generator architecture with a hierarchical multi-scale Gaussian representation that effectively regularizes the position and scale of generated Gaussians. Experimental results demonstrate that ours achieves a significantly faster rendering speed (x100) compared to state-of-the-art 3D consistent GANs.
  arXiv  Detail & Related papers  (2024-06-05T05:52:20Z)
• SuperGaussian: Repurposing Video Models for 3D Super Resolution [67.19266415499139]
  We present a simple, modular, and generic method that upsamples coarse 3D models by adding geometric and appearance details. We demonstrate that it is possible to directly repurpose existing (pretrained) video models for 3D super-resolution.
  arXiv  Detail & Related papers  (2024-06-02T03:44:50Z)
• GVGEN: Text-to-3D Generation with Volumetric Representation [89.55687129165256]
  3D Gaussian splatting has emerged as a powerful technique for 3D reconstruction and generation, known for its fast and high-quality rendering capabilities. This paper introduces a novel diffusion-based framework, GVGEN, designed to efficiently generate 3D Gaussian representations from text input.
  arXiv  Detail & Related papers  (2024-03-19T17:57:52Z)
• Hyper-3DG: Text-to-3D Gaussian Generation via Hypergraph [20.488040789522604]
  We propose a method named 3D Gaussian Generation via Hypergraph (Hyper-3DG)'', designed to capture the sophisticated high-order correlations present within 3D objects. Our framework allows for the production of finely generated 3D objects within a cohesive optimization, effectively circumventing degradation.
  arXiv  Detail & Related papers  (2024-03-14T09:59:55Z)
• GaussianDreamer: Fast Generation from Text to 3D Gaussians by Bridging 2D and 3D Diffusion Models [102.22388340738536]
  2D and 3D diffusion models can generate decent 3D objects based on prompts. 3D diffusion models have good 3D consistency, but their quality and generalization are limited as trainable 3D data is expensive and hard to obtain. This paper attempts to bridge the power from the two types of diffusion models via the recent explicit and efficient 3D Gaussian splatting representation.
  arXiv  Detail & Related papers  (2023-10-12T17:22:24Z)
• Text-to-3D using Gaussian Splatting [18.163413810199234]
  This paper proposes GSGEN, a novel method that adopts Gaussian Splatting, a recent state-of-the-art representation, to text-to-3D generation. GSGEN aims at generating high-quality 3D objects and addressing existing shortcomings by exploiting the explicit nature of Gaussian Splatting. Our approach can generate 3D assets with delicate details and accurate geometry.
  arXiv  Detail & Related papers  (2023-09-28T16:44:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.