CoNFies: Controllable Neural Face Avatars

• URL: http://arxiv.org/abs/2211.08610v1
• Date: Wed, 16 Nov 2022 01:43:43 GMT
• Title: CoNFies: Controllable Neural Face Avatars
• Authors: Heng Yu, Koichiro Niinuma, Laszlo A. Jeni
• Abstract summary: controllable neural representation for face self-portraits (CoNFies) We propose a controllable neural representation for face self-portraits (CoNFies) We use automated facial action recognition (AFAR) to characterize facial expressions as a combination of action units (AU) and their intensities.
• Score: 10.41057307836234
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Neural Radiance Fields (NeRF) are compelling techniques for modeling dynamic 3D scenes from 2D image collections. These volumetric representations would be well suited for synthesizing novel facial expressions but for two problems. First, deformable NeRFs are object agnostic and model holistic movement of the scene: they can replay how the motion changes over time, but they cannot alter it in an interpretable way. Second, controllable volumetric representations typically require either time-consuming manual annotations or 3D supervision to provide semantic meaning to the scene. We propose a controllable neural representation for face self-portraits (CoNFies), that solves both of these problems within a common framework, and it can rely on automated processing. We use automated facial action recognition (AFAR) to characterize facial expressions as a combination of action units (AU) and their intensities. AUs provide both the semantic locations and control labels for the system. CoNFies outperformed competing methods for novel view and expression synthesis in terms of visual and anatomic fidelity of expressions.

Related papers

• Cafca: High-quality Novel View Synthesis of Expressive Faces from Casual Few-shot Captures [33.463245327698]
  We present a novel volumetric prior on human faces that allows for high-fidelity expressive face modeling. We leverage a 3D Morphable Face Model to synthesize a large training set, rendering each identity with different expressions. We then train a conditional Neural Radiance Field prior on this synthetic dataset and, at inference time, fine-tune the model on a very sparse set of real images of a single subject.
  arXiv  Detail & Related papers  (2024-10-01T12:24:50Z)
• Pose-Controllable 3D Facial Animation Synthesis using Hierarchical Audio-Vertex Attention [52.63080543011595]
  A novel pose-controllable 3D facial animation synthesis method is proposed by utilizing hierarchical audio-vertex attention. The proposed method can produce more realistic facial expressions and head posture movements.
  arXiv  Detail & Related papers  (2023-02-24T09:36:31Z)
• Controllable Radiance Fields for Dynamic Face Synthesis [125.48602100893845]
  We study how to explicitly control generative model synthesis of face dynamics exhibiting non-rigid motion. Controllable Radiance Field (CoRF) On head image/video data we show that CoRFs are 3D-aware while enabling editing of identity, viewing directions, and motion.
  arXiv  Detail & Related papers  (2022-10-11T23:17:31Z)
• Neural Groundplans: Persistent Neural Scene Representations from a Single Image [90.04272671464238]
  We present a method to map 2D image observations of a scene to a persistent 3D scene representation. We propose conditional neural groundplans as persistent and memory-efficient scene representations.
  arXiv  Detail & Related papers  (2022-07-22T17:41:24Z)
• Control-NeRF: Editable Feature Volumes for Scene Rendering and Manipulation [58.16911861917018]
  We present a novel method for performing flexible, 3D-aware image content manipulation while enabling high-quality novel view synthesis. Our model couples learnt scene-specific feature volumes with a scene agnostic neural rendering network. We demonstrate various scene manipulations, including mixing scenes, deforming objects and inserting objects into scenes, while still producing photo-realistic results.
  arXiv  Detail & Related papers  (2022-04-22T17:57:00Z)
• FLAME-in-NeRF : Neural control of Radiance Fields for Free View Face Animation [37.39945646282971]
  This paper presents a neural rendering method for controllable portrait video synthesis. We leverage the expression space of a 3D morphable face model (3DMM) to represent the distribution of human facial expressions. We demonstrate the effectiveness of our method on free view synthesis of portrait videos with photorealistic expression controls.
  arXiv  Detail & Related papers  (2021-08-10T20:41:15Z)
• Neural Actor: Neural Free-view Synthesis of Human Actors with Pose Control [80.79820002330457]
  We propose a new method for high-quality synthesis of humans from arbitrary viewpoints and under arbitrary controllable poses. Our method achieves better quality than the state-of-the-arts on playback as well as novel pose synthesis, and can even generalize well to new poses that starkly differ from the training poses.
  arXiv  Detail & Related papers  (2021-06-03T17:40:48Z)
• Neural Monocular 3D Human Motion Capture with Physical Awareness [76.55971509794598]
  We present a new trainable system for physically plausible markerless 3D human motion capture. Unlike most neural methods for human motion capture, our approach is aware of physical and environmental constraints. It produces smooth and physically principled 3D motions in an interactive frame rate in a wide variety of challenging scenes.
  arXiv  Detail & Related papers  (2021-05-03T17:57:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.