Dynamic Neural Radiance Fields for Monocular 4D Facial Avatar Reconstruction

• URL: http://arxiv.org/abs/2012.03065v1
• Date: Sat, 5 Dec 2020 16:01:16 GMT
• Title: Dynamic Neural Radiance Fields for Monocular 4D Facial Avatar Reconstruction
• Authors: Guy Gafni, Justus Thies, Michael Zollh\"ofer, Matthias Nie{\ss}ner
• Abstract summary: We present dynamic neural radiance fields for modeling the appearance and dynamics of a human face. Especially, for telepresence applications in AR or VR, a faithful reproduction of the appearance including novel viewpoints or head-poses is required.
• Score: 9.747648609960185
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present dynamic neural radiance fields for modeling the appearance and dynamics of a human face. Digitally modeling and reconstructing a talking human is a key building-block for a variety of applications. Especially, for telepresence applications in AR or VR, a faithful reproduction of the appearance including novel viewpoints or head-poses is required. In contrast to state-of-the-art approaches that model the geometry and material properties explicitly, or are purely image-based, we introduce an implicit representation of the head based on scene representation networks. To handle the dynamics of the face, we combine our scene representation network with a low-dimensional morphable model which provides explicit control over pose and expressions. We use volumetric rendering to generate images from this hybrid representation and demonstrate that such a dynamic neural scene representation can be learned from monocular input data only, without the need of a specialized capture setup. In our experiments, we show that this learned volumetric representation allows for photo-realistic image generation that surpasses the quality of state-of-the-art video-based reenactment methods.

Related papers

• GaussianHeads: End-to-End Learning of Drivable Gaussian Head Avatars from Coarse-to-fine Representations [54.94362657501809]
  We propose a new method to generate highly dynamic and deformable human head avatars from multi-view imagery in real-time. At the core of our method is a hierarchical representation of head models that allows to capture the complex dynamics of facial expressions and head movements. We train this coarse-to-fine facial avatar model along with the head pose as a learnable parameter in an end-to-end framework.
  arXiv  Detail & Related papers  (2024-09-18T13:05:43Z)
• From Audio to Photoreal Embodiment: Synthesizing Humans in Conversations [107.88375243135579]
  Given speech audio, we output multiple possibilities of gestural motion for an individual, including face, body, and hands. We visualize the generated motion using highly photorealistic avatars that can express crucial nuances in gestures. Experiments show our model generates appropriate and diverse gestures, outperforming both diffusion- and VQ-only methods.
  arXiv  Detail & Related papers  (2024-01-03T18:55:16Z)
• Dynamic Multi-View Scene Reconstruction Using Neural Implicit Surface [0.9134661726886928]
  We propose a template-free method to reconstruct surface geometry and appearance using neural implicit representations from multi-view videos. We leverage topology-aware deformation and the signed distance field to learn complex dynamic surfaces via differentiable volume rendering. Experiments on different multi-view video datasets demonstrate that our method achieves high-fidelity surface reconstruction as well as photorealistic novel view synthesis.
  arXiv  Detail & Related papers  (2023-02-28T19:47:30Z)
• Instant Volumetric Head Avatars [20.782425305421505]
  We present Instant Volumetric Head Avatars (INSTA), a novel approach for reconstructing photo-realistic digital avatars instantaneously. Our pipeline is trained on a single monocular RGB portrait video that observes the subject under different expressions and views. INSTA reconstructs a digital avatar in less than 10 minutes on modern GPU hardware, which is orders of magnitude faster than previous solutions.
  arXiv  Detail & Related papers  (2022-11-22T18:59:46Z)
• I M Avatar: Implicit Morphable Head Avatars from Videos [68.13409777995392]
  We propose IMavatar, a novel method for learning implicit head avatars from monocular videos. Inspired by the fine-grained control mechanisms afforded by conventional 3DMMs, we represent the expression- and pose-related deformations via learned blendshapes and skinning fields. We show quantitatively and qualitatively that our method improves geometry and covers a more complete expression space compared to state-of-the-art methods.
  arXiv  Detail & Related papers  (2021-12-14T15:30:32Z)
• Neural Head Avatars from Monocular RGB Videos [0.0]
  We present a novel neural representation that explicitly models the surface geometry and appearance of an animatable human avatar. Our representation can be learned from a monocular RGB portrait video that features a range of different expressions and views.
  arXiv  Detail & Related papers  (2021-12-02T19:01:05Z)
• FLAME-in-NeRF : Neural control of Radiance Fields for Free View Face Animation [37.39945646282971]
  This paper presents a neural rendering method for controllable portrait video synthesis. We leverage the expression space of a 3D morphable face model (3DMM) to represent the distribution of human facial expressions. We demonstrate the effectiveness of our method on free view synthesis of portrait videos with photorealistic expression controls.
  arXiv  Detail & Related papers  (2021-08-10T20:41:15Z)
• Animatable Neural Radiance Fields from Monocular RGB Video [72.6101766407013]
  We present animatable neural radiance fields for detailed human avatar creation from monocular videos. Our approach extends neural radiance fields to the dynamic scenes with human movements via introducing explicit pose-guided deformation. In experiments we show that the proposed approach achieves 1) implicit human geometry and appearance reconstruction with high-quality details, 2) photo-realistic rendering of the human from arbitrary views, and 3) animation of the human with arbitrary poses.
  arXiv  Detail & Related papers  (2021-06-25T13:32:23Z)
• Neural Actor: Neural Free-view Synthesis of Human Actors with Pose Control [80.79820002330457]
  We propose a new method for high-quality synthesis of humans from arbitrary viewpoints and under arbitrary controllable poses. Our method achieves better quality than the state-of-the-arts on playback as well as novel pose synthesis, and can even generalize well to new poses that starkly differ from the training poses.
  arXiv  Detail & Related papers  (2021-06-03T17:40:48Z)
• D-NeRF: Neural Radiance Fields for Dynamic Scenes [72.75686949608624]
  We introduce D-NeRF, a method that extends neural radiance fields to a dynamic domain. D-NeRF reconstructs images of objects under rigid and non-rigid motions from a camera moving around the scene. We demonstrate the effectiveness of our approach on scenes with objects under rigid, articulated and non-rigid motions.
  arXiv  Detail & Related papers  (2020-11-27T19:06:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.