NeRFFaceEditing: Disentangled Face Editing in Neural Radiance Fields

• URL: http://arxiv.org/abs/2211.07968v1
• Date: Tue, 15 Nov 2022 08:11:39 GMT
• Title: NeRFFaceEditing: Disentangled Face Editing in Neural Radiance Fields
• Authors: Kaiwen Jiang, Shu-Yu Chen, Feng-Lin Liu, Hongbo Fu, Lin Gao
• Abstract summary: We introduce NeRFFaceEditing, which enables editing and decoupling geometry and appearance in neural radiance fields. Our method allows users to edit via semantic masks with decoupled control of geometry and appearance. Both qualitative and quantitative evaluations show the superior geometry and appearance control abilities of our method compared to existing and alternative solutions.
• Score: 40.543998582101146
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent methods for synthesizing 3D-aware face images have achieved rapid development thanks to neural radiance fields, allowing for high quality and fast inference speed. However, existing solutions for editing facial geometry and appearance independently usually require retraining and are not optimized for the recent work of generation, thus tending to lag behind the generation process. To address these issues, we introduce NeRFFaceEditing, which enables editing and decoupling geometry and appearance in the pretrained tri-plane-based neural radiance field while retaining its high quality and fast inference speed. Our key idea for disentanglement is to use the statistics of the tri-plane to represent the high-level appearance of its corresponding facial volume. Moreover, we leverage a generated 3D-continuous semantic mask as an intermediary for geometry editing. We devise a geometry decoder (whose output is unchanged when the appearance changes) and an appearance decoder. The geometry decoder aligns the original facial volume with the semantic mask volume. We also enhance the disentanglement by explicitly regularizing rendered images with the same appearance but different geometry to be similar in terms of color distribution for each facial component separately. Our method allows users to edit via semantic masks with decoupled control of geometry and appearance. Both qualitative and quantitative evaluations show the superior geometry and appearance control abilities of our method compared to existing and alternative solutions.

Related papers

• Revealing Directions for Text-guided 3D Face Editing [52.85632020601518]
  3D face editing is a significant task in multimedia, aimed at the manipulation of 3D face models across various control signals. We present Face Clan, a text-general approach for generating and manipulating 3D faces based on arbitrary attribute descriptions. Our method offers a precisely controllable manipulation method, allowing users to intuitively customize regions of interest with the text description.
  arXiv  Detail & Related papers  (2024-10-07T12:04:39Z)
• 3D Facial Expressions through Analysis-by-Neural-Synthesis [30.2749903946587]
  SMIRK (Spatial Modeling for Image-based Reconstruction of Kinesics) faithfully reconstructs expressive 3D faces from images. We identify two key limitations in existing methods: shortcomings in their self-supervised training formulation, and a lack of expression diversity in the training images. Our qualitative, quantitative and particularly our perceptual evaluations demonstrate that SMIRK achieves the new state-of-the art performance on accurate expression reconstruction.
  arXiv  Detail & Related papers  (2024-04-05T14:00:07Z)
• Text-Guided 3D Face Synthesis -- From Generation to Editing [53.86765812392627]
  We propose a unified text-guided framework from face generation to editing. We employ a fine-tuned texture diffusion model to enhance texture quality in both RGB and YUV space. We propose a self-guided consistency weight strategy to improve editing efficacy while preserving consistency.
  arXiv  Detail & Related papers  (2023-12-01T06:36:23Z)
• Neural Impostor: Editing Neural Radiance Fields with Explicit Shape Manipulation [49.852533321916844]
  We introduce Neural Impostor, a hybrid representation incorporating an explicit tetrahedral mesh alongside a multigrid implicit field. Our framework bridges the explicit shape manipulation and the geometric editing of implicit fields by utilizing multigrid barycentric coordinate encoding. We show the robustness and adaptability of our system through diverse examples and experiments, including the editing of both synthetic objects and real captured data.
  arXiv  Detail & Related papers  (2023-10-09T04:07:00Z)
• Delicate Textured Mesh Recovery from NeRF via Adaptive Surface Refinement [78.48648360358193]
  We present a novel framework that generates textured surface meshes from images. Our approach begins by efficiently initializing the geometry and view-dependency appearance with a NeRF. We jointly refine the appearance with geometry and bake it into texture images for real-time rendering.
  arXiv  Detail & Related papers  (2023-03-03T17:14:44Z)
• 3D Neural Sculpting (3DNS): Editing Neural Signed Distance Functions [34.39282814876276]
  In this work, we propose the first method for efficient interactive editing of signed distance functions expressed through neural networks. Inspired by 3D sculpting software for meshes, we use a brush-based framework that is intuitive and can in the future be used by sculptors and digital artists.
  arXiv  Detail & Related papers  (2022-09-28T10:05:16Z)
• IDE-3D: Interactive Disentangled Editing for High-Resolution 3D-aware Portrait Synthesis [38.517819699560945]
  Our system consists of three major components: (1) a 3D-semantics-aware generative model that produces view-consistent, disentangled face images and semantic masks; (2) a hybrid GAN inversion approach that initializes the latent codes from the semantic and texture encoder, and further optimized them for faithful reconstruction; and (3) a canonical editor that enables efficient manipulation of semantic masks in canonical view and product high-quality editing results.
  arXiv  Detail & Related papers  (2022-05-31T03:35:44Z)
• Beyond 3DMM: Learning to Capture High-fidelity 3D Face Shape [77.95154911528365]
  3D Morphable Model (3DMM) fitting has widely benefited face analysis due to its strong 3D priori. Previous reconstructed 3D faces suffer from degraded visual verisimilitude due to the loss of fine-grained geometry. This paper proposes a complete solution to capture the personalized shape so that the reconstructed shape looks identical to the corresponding person.
  arXiv  Detail & Related papers  (2022-04-09T03:46:18Z)
• Topologically Consistent Multi-View Face Inference Using Volumetric Sampling [25.001398662643986]
  ToFu is a geometry inference framework that can produce topologically consistent meshes across identities and expressions. A novel progressive mesh generation network embeds the topological structure of the face in a feature volume. These high-quality assets are readily usable by production studios for avatar creation, animation and physically-based skin rendering.
  arXiv  Detail & Related papers  (2021-10-06T17:55:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.