MoFaNeRF: Morphable Facial Neural Radiance Field

• URL: http://arxiv.org/abs/2112.02308v1
• Date: Sat, 4 Dec 2021 11:25:28 GMT
• Title: MoFaNeRF: Morphable Facial Neural Radiance Field
• Authors: Yiyu Zhuang, Hao Zhu, Xusen Sun, Xun Cao
• Abstract summary: MoFaNeRF is a parametric model that maps free-view images into a vector space coded facial shape, expression and appearance. By introducing identity-specific modulation and encoder texture, our model synthesizes accurate photometric details. Our model shows strong ability on multiple applications including image-based fitting, random generation, face rigging, face editing, and novel view.
• Score: 12.443638713719357
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We propose a parametric model that maps free-view images into a vector space of coded facial shape, expression and appearance using a neural radiance field, namely Morphable Facial NeRF. Specifically, MoFaNeRF takes the coded facial shape, expression and appearance along with space coordinate and view direction as input to an MLP, and outputs the radiance of the space point for photo-realistic image synthesis. Compared with conventional 3D morphable models (3DMM), MoFaNeRF shows superiority in directly synthesizing photo-realistic facial details even for eyes, mouths, and beards. Also, continuous face morphing can be easily achieved by interpolating the input shape, expression and appearance codes. By introducing identity-specific modulation and texture encoder, our model synthesizes accurate photometric details and shows strong representation ability. Our model shows strong ability on multiple applications including image-based fitting, random generation, face rigging, face editing, and novel view synthesis. Experiments show that our method achieves higher representation ability than previous parametric models, and achieves competitive performance in several applications. To the best of our knowledge, our work is the first facial parametric model built upon a neural radiance field that can be used in fitting, generation and manipulation. Our code and model are released in https://github.com/zhuhao-nju/mofanerf.

Related papers

• Cafca: High-quality Novel View Synthesis of Expressive Faces from Casual Few-shot Captures [33.463245327698]
  We present a novel volumetric prior on human faces that allows for high-fidelity expressive face modeling. We leverage a 3D Morphable Face Model to synthesize a large training set, rendering each identity with different expressions. We then train a conditional Neural Radiance Field prior on this synthetic dataset and, at inference time, fine-tune the model on a very sparse set of real images of a single subject.
  arXiv  Detail & Related papers  (2024-10-01T12:24:50Z)
• Single-Shot Implicit Morphable Faces with Consistent Texture Parameterization [91.52882218901627]
  We propose a novel method for constructing implicit 3D morphable face models that are both generalizable and intuitive for editing. Our method improves upon photo-realism, geometry, and expression accuracy compared to state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-05-04T17:58:40Z)
• One-Shot High-Fidelity Talking-Head Synthesis with Deformable Neural Radiance Field [81.07651217942679]
  Talking head generation aims to generate faces that maintain the identity information of the source image and imitate the motion of the driving image. We propose HiDe-NeRF, which achieves high-fidelity and free-view talking-head synthesis.
  arXiv  Detail & Related papers  (2023-04-11T09:47:35Z)
• Reconstructing Personalized Semantic Facial NeRF Models From Monocular Video [27.36067360218281]
  We present a novel semantic model for human head defined with neural radiance field. The 3D-consistent head model consist of a set of disentangled and interpretable bases, and can be driven by low-dimensional expression coefficients. With a short monocular RGB video as input, our method can construct the subject's semantic facial NeRF model with only ten to twenty minutes.
  arXiv  Detail & Related papers  (2022-10-12T11:56:52Z)
• 3DMM-RF: Convolutional Radiance Fields for 3D Face Modeling [111.98096975078158]
  We introduce a style-based generative network that synthesizes in one pass all and only the required rendering samples of a neural radiance field. We show that this model can accurately be fit to "in-the-wild" facial images of arbitrary pose and illumination, extract the facial characteristics, and be used to re-render the face in controllable conditions.
  arXiv  Detail & Related papers  (2022-09-15T15:28:45Z)
• Learning Dynamic Facial Radiance Fields for Few-Shot Talking Head Synthesis [90.43371339871105]
  We propose Dynamic Facial Radiance Fields (DFRF) for few-shot talking head synthesis. DFRF conditions face radiance field on 2D appearance images to learn the face prior. Experiments show DFRF can synthesize natural and high-quality audio-driven talking head videos for novel identities with only 40k iterations.
  arXiv  Detail & Related papers  (2022-07-24T16:46:03Z)
• Shape My Face: Registering 3D Face Scans by Surface-to-Surface Translation [75.59415852802958]
  Shape-My-Face (SMF) is a powerful encoder-decoder architecture based on an improved point cloud encoder, a novel visual attention mechanism, graph convolutional decoders with skip connections, and a specialized mouth model. Our model provides topologically-sound meshes with minimal supervision, offers faster training time, has orders of magnitude fewer trainable parameters, is more robust to noise, and can generalize to previously unseen datasets.
  arXiv  Detail & Related papers  (2020-12-16T20:02:36Z)
• Portrait Neural Radiance Fields from a Single Image [68.66958204066721]
  We present a method for estimating Neural Radiance Fields (NeRF) from a single portrait. We propose to pretrain the weights of a multilayer perceptron (MLP), which implicitly models the volumetric density. To improve the generalization to unseen faces, we train the canonical coordinate space approximated by 3D face morphable models. We quantitatively evaluate the method using controlled captures and demonstrate the generalization to real portrait images, showing favorable results against state-of-the-arts.
  arXiv  Detail & Related papers  (2020-12-10T18:59:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.