Training and Tuning Generative Neural Radiance Fields for
Attribute-Conditional 3D-Aware Face Generation
- URL: http://arxiv.org/abs/2208.12550v2
- Date: Wed, 18 Oct 2023 04:00:45 GMT
- Title: Training and Tuning Generative Neural Radiance Fields for
Attribute-Conditional 3D-Aware Face Generation
- Authors: Jichao Zhang, Aliaksandr Siarohin, Yahui Liu, Hao Tang, Nicu Sebe, Wei
Wang
- Abstract summary: Generative Neural Radiance Fields (GNeRF) based 3D-aware GANs have demonstrated remarkable capabilities in generating high-quality images.
We propose a conditional GNeRF model incorporating specific attribute labels as input to enhance the controllability and disentanglement abilities of 3D-aware generative models.
- Score: 69.53142666853502
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Generative Neural Radiance Fields (GNeRF) based 3D-aware GANs have
demonstrated remarkable capabilities in generating high-quality images while
maintaining strong 3D consistency. Notably, significant advancements have been
made in the domain of face generation. However, most existing models prioritize
view consistency over disentanglement, resulting in limited semantic/attribute
control during generation. To address this limitation, we propose a conditional
GNeRF model incorporating specific attribute labels as input to enhance the
controllability and disentanglement abilities of 3D-aware generative models.
Our approach builds upon a pre-trained 3D-aware face model, and we introduce a
Training as Init and Optimizing for Tuning (TRIOT) method to train a
conditional normalized flow module to enable the facial attribute editing, then
optimize the latent vector to improve attribute-editing precision further. Our
extensive experiments demonstrate that our model produces high-quality edits
with superior view consistency while preserving non-target regions. Code is
available at https://github.com/zhangqianhui/TT-GNeRF.
Related papers
- Controllable Text-to-3D Generation via Surface-Aligned Gaussian Splatting [9.383423119196408]
We introduce Multi-view ControlNet (MVControl), a novel neural network architecture designed to enhance existing multi-view diffusion models.
MVControl is able to offer 3D diffusion guidance for optimization-based 3D generation.
In pursuit of efficiency, we adopt 3D Gaussians as our representation instead of the commonly used implicit representations.
arXiv Detail & Related papers (2024-03-15T02:57:20Z) - Spec-Gaussian: Anisotropic View-Dependent Appearance for 3D Gaussian
Splatting [57.80942520483354]
3D-GS frequently encounters difficulties in accurately modeling specular and anisotropic components.
We introduce Spec-Gaussian, an approach that utilizes an anisotropic spherical Gaussian appearance field instead of spherical harmonics.
Our experimental results demonstrate that our method surpasses existing approaches in terms of rendering quality.
arXiv Detail & Related papers (2024-02-24T17:22:15Z) - Pushing Auto-regressive Models for 3D Shape Generation at Capacity and Scalability [118.26563926533517]
Auto-regressive models have achieved impressive results in 2D image generation by modeling joint distributions in grid space.
We extend auto-regressive models to 3D domains, and seek a stronger ability of 3D shape generation by improving auto-regressive models at capacity and scalability simultaneously.
arXiv Detail & Related papers (2024-02-19T15:33:09Z) - Enhancing High-Resolution 3D Generation through Pixel-wise Gradient
Clipping [46.364968008574664]
High-resolution 3D object generation remains a challenging task due to limited availability of comprehensive annotated training data.
Recent advancements have aimed to overcome this constraint by harnessing image generative models, pretrained on extensive curated web datasets.
We propose an innovative operation termed Pixel-wise Gradient Clipping (PGC) designed for seamless integration into existing 3D generative models.
arXiv Detail & Related papers (2023-10-19T05:15:17Z) - Learning Versatile 3D Shape Generation with Improved AR Models [91.87115744375052]
Auto-regressive (AR) models have achieved impressive results in 2D image generation by modeling joint distributions in the grid space.
We propose the Improved Auto-regressive Model (ImAM) for 3D shape generation, which applies discrete representation learning based on a latent vector instead of volumetric grids.
arXiv Detail & Related papers (2023-03-26T12:03:18Z) - NeRF-GAN Distillation for Efficient 3D-Aware Generation with
Convolutions [97.27105725738016]
integration of Neural Radiance Fields (NeRFs) and generative models, such as Generative Adversarial Networks (GANs) has transformed 3D-aware generation from single-view images.
We propose a simple and effective method, based on re-using the well-disentangled latent space of a pre-trained NeRF-GAN in a pose-conditioned convolutional network to directly generate 3D-consistent images corresponding to the underlying 3D representations.
arXiv Detail & Related papers (2023-03-22T18:59:48Z) - 3D Generative Model Latent Disentanglement via Local Eigenprojection [13.713373496487012]
We introduce a novel loss function grounded in spectral geometry for different neural-network-based generative models of 3D head and body meshes.
Experimental results show that our local eigenprojection disentangled (LED) models offer improved disentanglement with respect to the state-of-the-art.
arXiv Detail & Related papers (2023-02-24T18:19:49Z) - CGOF++: Controllable 3D Face Synthesis with Conditional Generative
Occupancy Fields [52.14985242487535]
We propose a new conditional 3D face synthesis framework, which enables 3D controllability over generated face images.
At its core is a conditional Generative Occupancy Field (cGOF++) that effectively enforces the shape of the generated face to conform to a given 3D Morphable Model (3DMM) mesh.
Experiments validate the effectiveness of the proposed method and show more precise 3D controllability than state-of-the-art 2D-based controllable face synthesis methods.
arXiv Detail & Related papers (2022-11-23T19:02:50Z) - AE-NeRF: Auto-Encoding Neural Radiance Fields for 3D-Aware Object
Manipulation [24.65896451569795]
We propose a novel framework for 3D-aware object manipulation, called Auto-aware Neural Radiance Fields (AE-NeRF)
Our model is formulated in an auto-encoder architecture, extracts disentangled 3D attributes such as 3D shape, appearance, and camera pose from an image.
A high-quality image is rendered from the attributes through disentangled generative Neural Radiance Fields (NeRF)
arXiv Detail & Related papers (2022-04-28T11:50:18Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.