Training and Tuning Generative Neural Radiance Fields for Attribute-Conditional 3D-Aware Face Generation

• URL: http://arxiv.org/abs/2208.12550v3
• Date: Sun, 1 Sep 2024 06:14:37 GMT
• Title: Training and Tuning Generative Neural Radiance Fields for Attribute-Conditional 3D-Aware Face Generation
• Authors: Jichao Zhang, Aliaksandr Siarohin, Yahui Liu, Hao Tang, Nicu Sebe, Wei Wang,
• Abstract summary: We propose a conditional GNeRF model that integrates specific attribute labels as input, thus amplifying the controllability and disentanglement capabilities of 3D-aware generative models. Our approach builds upon a pre-trained 3D-aware face model, and we introduce a Training as Init and fidelity for Tuning (TRIOT) method to train a conditional normalized flow module. Our experiments substantiate the efficacy of our model, showcasing its ability to generate high-quality edits with enhanced view consistency.
• Score: 66.21121745446345
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Generative Neural Radiance Fields (GNeRF)-based 3D-aware GANs have showcased remarkable prowess in crafting high-fidelity images while upholding robust 3D consistency, particularly face generation. However, specific existing models prioritize view consistency over disentanglement, leading to constrained semantic or attribute control during the generation process. While many methods have explored incorporating semantic masks or leveraging 3D Morphable Models (3DMM) priors to imbue models with semantic control, these methods often demand training from scratch, entailing significant computational overhead. In this paper, we propose a novel approach: a conditional GNeRF model that integrates specific attribute labels as input, thus amplifying the controllability and disentanglement capabilities of 3D-aware generative models. Our approach builds upon a pre-trained 3D-aware face model, and we introduce a Training as Init and Optimizing for Tuning (TRIOT) method to train a conditional normalized flow module to enable the facial attribute editing, then optimize the latent vector to improve attribute-editing precision further. Our extensive experiments substantiate the efficacy of our model, showcasing its ability to generate high-quality edits with enhanced view consistency while safeguarding non-target regions. The code for our model is publicly available at https://github.com/zhangqianhui/TT-GNeRF.

Related papers

• TripoSG: High-Fidelity 3D Shape Synthesis using Large-Scale Rectified Flow Models [69.0220314849478]
  TripoSG is a new paradigm capable of generating high-fidelity 3D meshes with precise correspondence to input images. The resulting 3D shapes exhibit en- hanced detail due to high-resolution capabilities and demonstrate exceptional fidelity to input im- ages. To foster progress and innovation in the field of 3D generation, we will make our model publicly available.
  arXiv  Detail & Related papers  (2025-02-10T16:07:54Z)
• A Lesson in Splats: Teacher-Guided Diffusion for 3D Gaussian Splats Generation with 2D Supervision [65.33043028101471]
  We introduce a diffusion model for Gaussian Splats, SplatDiffusion, to enable generation of three-dimensional structures from single images. Existing methods rely on deterministic, feed-forward predictions, which limit their ability to handle the inherent ambiguity of 3D inference from 2D data.
  arXiv  Detail & Related papers  (2024-12-01T00:29:57Z)
• 3D-WAG: Hierarchical Wavelet-Guided Autoregressive Generation for High-Fidelity 3D Shapes [20.675695749508353]
  We introduce 3D-WAG, an AR model for 3D implicit distance fields that can perform unconditional shape generation. By redefining 3D AR generation task as next-scale" prediction, we reduce the computational cost of generation. Our results show 3D-WAG achieves superior performance in key metrics like Coverage and MMD.
  arXiv  Detail & Related papers  (2024-11-28T10:33:01Z)
• Masked Generative Extractor for Synergistic Representation and 3D Generation of Point Clouds [6.69660410213287]
  We propose an innovative framework called Point-MGE to explore the benefits of deeply integrating 3D representation learning and generative learning. In shape classification, Point-MGE achieved an accuracy of 94.2% (+1.0%) on the ModelNet40 dataset and 92.9% (+5.5%) on the ScanObjectNN dataset. Experimental results also confirmed that Point-MGE can generate high-quality 3D shapes in both unconditional and conditional settings.
  arXiv  Detail & Related papers  (2024-06-25T07:57:03Z)
• MeshXL: Neural Coordinate Field for Generative 3D Foundation Models [51.1972329762843]
  We present a family of generative pre-trained auto-regressive models, which addresses the process of 3D mesh generation with modern large language model approaches. MeshXL is able to generate high-quality 3D meshes, and can also serve as foundation models for various down-stream applications.
  arXiv  Detail & Related papers  (2024-05-31T14:35:35Z)
• Controllable Text-to-3D Generation via Surface-Aligned Gaussian Splatting [9.383423119196408]
  We introduce Multi-view ControlNet (MVControl), a novel neural network architecture designed to enhance existing multi-view diffusion models. MVControl is able to offer 3D diffusion guidance for optimization-based 3D generation. In pursuit of efficiency, we adopt 3D Gaussians as our representation instead of the commonly used implicit representations.
  arXiv  Detail & Related papers  (2024-03-15T02:57:20Z)
• Learning Versatile 3D Shape Generation with Improved AR Models [91.87115744375052]
  Auto-regressive (AR) models have achieved impressive results in 2D image generation by modeling joint distributions in the grid space. We propose the Improved Auto-regressive Model (ImAM) for 3D shape generation, which applies discrete representation learning based on a latent vector instead of volumetric grids.
  arXiv  Detail & Related papers  (2023-03-26T12:03:18Z)
• NeRF-GAN Distillation for Efficient 3D-Aware Generation with Convolutions [97.27105725738016]
  integration of Neural Radiance Fields (NeRFs) and generative models, such as Generative Adversarial Networks (GANs) has transformed 3D-aware generation from single-view images. We propose a simple and effective method, based on re-using the well-disentangled latent space of a pre-trained NeRF-GAN in a pose-conditioned convolutional network to directly generate 3D-consistent images corresponding to the underlying 3D representations.
  arXiv  Detail & Related papers  (2023-03-22T18:59:48Z)
• 3D Generative Model Latent Disentanglement via Local Eigenprojection [13.713373496487012]
  We introduce a novel loss function grounded in spectral geometry for different neural-network-based generative models of 3D head and body meshes. Experimental results show that our local eigenprojection disentangled (LED) models offer improved disentanglement with respect to the state-of-the-art.
  arXiv  Detail & Related papers  (2023-02-24T18:19:49Z)
• AE-NeRF: Auto-Encoding Neural Radiance Fields for 3D-Aware Object Manipulation [24.65896451569795]
  We propose a novel framework for 3D-aware object manipulation, called Auto-aware Neural Radiance Fields (AE-NeRF) Our model is formulated in an auto-encoder architecture, extracts disentangled 3D attributes such as 3D shape, appearance, and camera pose from an image. A high-quality image is rendered from the attributes through disentangled generative Neural Radiance Fields (NeRF)
  arXiv  Detail & Related papers  (2022-04-28T11:50:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.