Reconstructing Personalized Semantic Facial NeRF Models From Monocular Video

• URL: http://arxiv.org/abs/2210.06108v1
• Date: Wed, 12 Oct 2022 11:56:52 GMT
• Title: Reconstructing Personalized Semantic Facial NeRF Models From Monocular Video
• Authors: Xuan Gao, Chenglai Zhong, Jun Xiang, Yang Hong, Yudong Guo, Juyong Zhang
• Abstract summary: We present a novel semantic model for human head defined with neural radiance field. The 3D-consistent head model consist of a set of disentangled and interpretable bases, and can be driven by low-dimensional expression coefficients. With a short monocular RGB video as input, our method can construct the subject's semantic facial NeRF model with only ten to twenty minutes.
• Score: 27.36067360218281
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present a novel semantic model for human head defined with neural radiance field. The 3D-consistent head model consist of a set of disentangled and interpretable bases, and can be driven by low-dimensional expression coefficients. Thanks to the powerful representation ability of neural radiance field, the constructed model can represent complex facial attributes including hair, wearings, which can not be represented by traditional mesh blendshape. To construct the personalized semantic facial model, we propose to define the bases as several multi-level voxel fields. With a short monocular RGB video as input, our method can construct the subject's semantic facial NeRF model with only ten to twenty minutes, and can render a photo-realistic human head image in tens of miliseconds with a given expression coefficient and view direction. With this novel representation, we apply it to many tasks like facial retargeting and expression editing. Experimental results demonstrate its strong representation ability and training/inference speed. Demo videos and released code are provided in our project page: https://ustc3dv.github.io/NeRFBlendShape/

Related papers

• DistillNeRF: Perceiving 3D Scenes from Single-Glance Images by Distilling Neural Fields and Foundation Model Features [65.8738034806085]
  DistillNeRF is a self-supervised learning framework for understanding 3D environments in autonomous driving scenes. Our method is a generalizable feedforward model that predicts a rich neural scene representation from sparse, single-frame multi-view camera inputs.
  arXiv  Detail & Related papers  (2024-06-17T21:15:13Z)
• 3DMM-RF: Convolutional Radiance Fields for 3D Face Modeling [111.98096975078158]
  We introduce a style-based generative network that synthesizes in one pass all and only the required rendering samples of a neural radiance field. We show that this model can accurately be fit to "in-the-wild" facial images of arbitrary pose and illumination, extract the facial characteristics, and be used to re-render the face in controllable conditions.
  arXiv  Detail & Related papers  (2022-09-15T15:28:45Z)
• Multi-NeuS: 3D Head Portraits from Single Image with Neural Implicit Functions [70.04394678730968]
  We present an approach for the reconstruction of 3D human heads from one or few views. The underlying neural architecture is to learn the objects and to generalize the model. Our model can fit novel heads on just a hundred videos or one-shot 3D scans.
  arXiv  Detail & Related papers  (2022-09-07T21:09:24Z)
• Learning Dynamic Facial Radiance Fields for Few-Shot Talking Head Synthesis [90.43371339871105]
  We propose Dynamic Facial Radiance Fields (DFRF) for few-shot talking head synthesis. DFRF conditions face radiance field on 2D appearance images to learn the face prior. Experiments show DFRF can synthesize natural and high-quality audio-driven talking head videos for novel identities with only 40k iterations.
  arXiv  Detail & Related papers  (2022-07-24T16:46:03Z)
• RigNeRF: Fully Controllable Neural 3D Portraits [52.91370717599413]
  RigNeRF enables full control of head pose and facial expressions learned from a single portrait video. We demonstrate the effectiveness of our method on free view synthesis of a portrait scene with explicit head pose and expression controls.
  arXiv  Detail & Related papers  (2022-06-13T21:28:34Z)
• Semantic-Aware Implicit Neural Audio-Driven Video Portrait Generation [61.8546794105462]
  We propose Semantic-aware Speaking Portrait NeRF (SSP-NeRF), which creates delicate audio-driven portraits using one unified set of NeRF. We first propose a Semantic-Aware Dynamic Ray Sampling module with an additional parsing branch that facilitates audio-driven volume rendering. To enable portrait rendering in one unified neural radiance field, a Torso Deformation module is designed to stabilize the large-scale non-rigid torso motions.
  arXiv  Detail & Related papers  (2022-01-19T18:54:41Z)
• MoFaNeRF: Morphable Facial Neural Radiance Field [12.443638713719357]
  MoFaNeRF is a parametric model that maps free-view images into a vector space coded facial shape, expression and appearance. By introducing identity-specific modulation and encoder texture, our model synthesizes accurate photometric details. Our model shows strong ability on multiple applications including image-based fitting, random generation, face rigging, face editing, and novel view.
  arXiv  Detail & Related papers  (2021-12-04T11:25:28Z)
• Neural Head Avatars from Monocular RGB Videos [0.0]
  We present a novel neural representation that explicitly models the surface geometry and appearance of an animatable human avatar. Our representation can be learned from a monocular RGB portrait video that features a range of different expressions and views.
  arXiv  Detail & Related papers  (2021-12-02T19:01:05Z)
• Dynamic Neural Radiance Fields for Monocular 4D Facial Avatar Reconstruction [9.747648609960185]
  We present dynamic neural radiance fields for modeling the appearance and dynamics of a human face. Especially, for telepresence applications in AR or VR, a faithful reproduction of the appearance including novel viewpoints or head-poses is required.
  arXiv  Detail & Related papers  (2020-12-05T16:01:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.