GGHead: Fast and Generalizable 3D Gaussian Heads

• URL: http://arxiv.org/abs/2406.09377v2
• Date: Tue, 24 Sep 2024 14:48:19 GMT
• Title: GGHead: Fast and Generalizable 3D Gaussian Heads
• Authors: Tobias Kirschstein, Simon Giebenhain, Jiapeng Tang, Markos Georgopoulos, Matthias Nießner,
• Abstract summary: 3D GANs struggle to scale to generate samples at high resolutions due to their relatively slow train and render speeds. We propose Generative Gaussian Heads (GGHead), which adopts the recent 3D Gaussian Splatting representation within a 3D GAN framework. We demonstrate real-time generation and rendering of high-quality 3D-consistent heads at $10242$ resolution for the first time.
• Score: 48.967905053963385
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Learning 3D head priors from large 2D image collections is an important step towards high-quality 3D-aware human modeling. A core requirement is an efficient architecture that scales well to large-scale datasets and large image resolutions. Unfortunately, existing 3D GANs struggle to scale to generate samples at high resolutions due to their relatively slow train and render speeds, and typically have to rely on 2D superresolution networks at the expense of global 3D consistency. To address these challenges, we propose Generative Gaussian Heads (GGHead), which adopts the recent 3D Gaussian Splatting representation within a 3D GAN framework. To generate a 3D representation, we employ a powerful 2D CNN generator to predict Gaussian attributes in the UV space of a template head mesh. This way, GGHead exploits the regularity of the template's UV layout, substantially facilitating the challenging task of predicting an unstructured set of 3D Gaussians. We further improve the geometric fidelity of the generated 3D representations with a novel total variation loss on rendered UV coordinates. Intuitively, this regularization encourages that neighboring rendered pixels should stem from neighboring Gaussians in the template's UV space. Taken together, our pipeline can efficiently generate 3D heads trained only from single-view 2D image observations. Our proposed framework matches the quality of existing 3D head GANs on FFHQ while being both substantially faster and fully 3D consistent. As a result, we demonstrate real-time generation and rendering of high-quality 3D-consistent heads at $1024^2$ resolution for the first time. Project Website: https://tobias-kirschstein.github.io/gghead

Related papers

• GSD: View-Guided Gaussian Splatting Diffusion for 3D Reconstruction [52.04103235260539]
  We present a diffusion model approach based on Gaussian Splatting representation for 3D object reconstruction from a single view. The model learns to generate 3D objects represented by sets of GS ellipsoids. The final reconstructed objects explicitly come with high-quality 3D structure and texture, and can be efficiently rendered in arbitrary views.
  arXiv  Detail & Related papers  (2024-07-05T03:43:08Z)
• DIRECT-3D: Learning Direct Text-to-3D Generation on Massive Noisy 3D Data [50.164670363633704]
  We present DIRECT-3D, a diffusion-based 3D generative model for creating high-quality 3D assets from text prompts. Our model is directly trained on extensive noisy and unaligned in-the-wild' 3D assets. We achieve state-of-the-art performance in both single-class generation and text-to-3D generation.
  arXiv  Detail & Related papers  (2024-06-06T17:58:15Z)
• Gaussian Splatting Decoder for 3D-aware Generative Adversarial Networks [10.207899254360374]
  NeRF-based 3D-aware Generative Adversarial Networks (GANs) have shown very high rendering quality under large representational variety. rendering with Neural Radiance Fields poses challenges for 3D applications. We present a novel approach that combines the high rendering quality of NeRF-based 3D-aware GANs with the flexibility and computational advantages of 3DGS.
  arXiv  Detail & Related papers  (2024-04-16T14:48:40Z)
• AGG: Amortized Generative 3D Gaussians for Single Image to 3D [108.38567665695027]
  We introduce an Amortized Generative 3D Gaussian framework (AGG) that instantly produces 3D Gaussians from a single image. AGG decomposes the generation of 3D Gaussian locations and other appearance attributes for joint optimization. We propose a cascaded pipeline that first generates a coarse representation of the 3D data and later upsamples it with a 3D Gaussian super-resolution module.
  arXiv  Detail & Related papers  (2024-01-08T18:56:33Z)
• GaussianDreamer: Fast Generation from Text to 3D Gaussians by Bridging 2D and 3D Diffusion Models [102.22388340738536]
  2D and 3D diffusion models can generate decent 3D objects based on prompts. 3D diffusion models have good 3D consistency, but their quality and generalization are limited as trainable 3D data is expensive and hard to obtain. This paper attempts to bridge the power from the two types of diffusion models via the recent explicit and efficient 3D Gaussian splatting representation.
  arXiv  Detail & Related papers  (2023-10-12T17:22:24Z)
• XDGAN: Multi-Modal 3D Shape Generation in 2D Space [60.46777591995821]
  We propose a novel method to convert 3D shapes into compact 1-channel geometry images and leverage StyleGAN3 and image-to-image translation networks to generate 3D objects in 2D space. The generated geometry images are quick to convert to 3D meshes, enabling real-time 3D object synthesis, visualization and interactive editing. We show both quantitatively and qualitatively that our method is highly effective at various tasks such as 3D shape generation, single view reconstruction and shape manipulation, while being significantly faster and more flexible compared to recent 3D generative models.
  arXiv  Detail & Related papers  (2022-10-06T15:54:01Z)
• GRAM-HD: 3D-Consistent Image Generation at High Resolution with Generative Radiance Manifolds [28.660893916203747]
  This paper proposes a novel 3D-aware GAN that can generate high resolution images (up to 1024X1024) while keeping strict 3D consistency as in volume rendering. Our motivation is to achieve super-resolution directly in the 3D space to preserve 3D consistency. Experiments on FFHQ and AFHQv2 datasets show that our method can produce high-quality 3D-consistent results.
  arXiv  Detail & Related papers  (2022-06-15T02:35:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.