Neural Head Avatars from Monocular RGB Videos

• URL: http://arxiv.org/abs/2112.01554v1
• Date: Thu, 2 Dec 2021 19:01:05 GMT
• Title: Neural Head Avatars from Monocular RGB Videos
• Authors: Philip-William Grassal (1), Malte Prinzler (1), Titus Leistner (1), Carsten Rother (1), Matthias Nie{\ss}ner (2), Justus Thies (3) ((1) Heidelberg University, (2) Technical University of Munich, (3) Max Planck Institute for Intelligent Systems)
• Abstract summary: We present a novel neural representation that explicitly models the surface geometry and appearance of an animatable human avatar. Our representation can be learned from a monocular RGB portrait video that features a range of different expressions and views.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present Neural Head Avatars, a novel neural representation that explicitly models the surface geometry and appearance of an animatable human avatar that can be used for teleconferencing in AR/VR or other applications in the movie or games industry that rely on a digital human. Our representation can be learned from a monocular RGB portrait video that features a range of different expressions and views. Specifically, we propose a hybrid representation consisting of a morphable model for the coarse shape and expressions of the face, and two feed-forward networks, predicting vertex offsets of the underlying mesh as well as a view- and expression-dependent texture. We demonstrate that this representation is able to accurately extrapolate to unseen poses and view points, and generates natural expressions while providing sharp texture details. Compared to previous works on head avatars, our method provides a disentangled shape and appearance model of the complete human head (including hair) that is compatible with the standard graphics pipeline. Moreover, it quantitatively and qualitatively outperforms current state of the art in terms of reconstruction quality and novel-view synthesis.

Related papers

• GaussianHeads: End-to-End Learning of Drivable Gaussian Head Avatars from Coarse-to-fine Representations [54.94362657501809]
  We propose a new method to generate highly dynamic and deformable human head avatars from multi-view imagery in real-time. At the core of our method is a hierarchical representation of head models that allows to capture the complex dynamics of facial expressions and head movements. We train this coarse-to-fine facial avatar model along with the head pose as a learnable parameter in an end-to-end framework.
  arXiv  Detail & Related papers  (2024-09-18T13:05:43Z)
• HR Human: Modeling Human Avatars with Triangular Mesh and High-Resolution Textures from Videos [52.23323966700072]
  We present a framework for acquiring human avatars that are attached with high-resolution physically-based material textures and mesh from monocular video. Our method introduces a novel information fusion strategy to combine the information from the monocular video and synthesize virtual multi-view images. Experiments show that our approach outperforms previous representations in terms of high fidelity, and this explicit result supports deployment on common triangulars.
  arXiv  Detail & Related papers  (2024-05-18T11:49:09Z)
• BakedAvatar: Baking Neural Fields for Real-Time Head Avatar Synthesis [7.485318043174123]
  We introduce BakedAvatar, a novel representation for real-time neural head avatar. Our approach extracts layered meshes from learned isosurfaces of the head and computes expression-, pose-, and view-dependent appearances. Experimental results demonstrate that our representation generates photorealistic results of comparable quality to other state-the-art methods.
  arXiv  Detail & Related papers  (2023-11-09T17:05:53Z)
• Single-Shot Implicit Morphable Faces with Consistent Texture Parameterization [91.52882218901627]
  We propose a novel method for constructing implicit 3D morphable face models that are both generalizable and intuitive for editing. Our method improves upon photo-realism, geometry, and expression accuracy compared to state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-05-04T17:58:40Z)
• AniPixel: Towards Animatable Pixel-Aligned Human Avatar [65.7175527782209]
  AniPixel is a novel animatable and generalizable human avatar reconstruction method. We propose a neural skinning field based on skeleton-driven deformation to establish the target-to-canonical and canonical-to-observation correspondences. Experiments show that AniPixel renders comparable novel views while delivering better novel pose animation results than state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-02-07T11:04:14Z)
• Reconstructing Personalized Semantic Facial NeRF Models From Monocular Video [27.36067360218281]
  We present a novel semantic model for human head defined with neural radiance field. The 3D-consistent head model consist of a set of disentangled and interpretable bases, and can be driven by low-dimensional expression coefficients. With a short monocular RGB video as input, our method can construct the subject's semantic facial NeRF model with only ten to twenty minutes.
  arXiv  Detail & Related papers  (2022-10-12T11:56:52Z)
• 3DMM-RF: Convolutional Radiance Fields for 3D Face Modeling [111.98096975078158]
  We introduce a style-based generative network that synthesizes in one pass all and only the required rendering samples of a neural radiance field. We show that this model can accurately be fit to "in-the-wild" facial images of arbitrary pose and illumination, extract the facial characteristics, and be used to re-render the face in controllable conditions.
  arXiv  Detail & Related papers  (2022-09-15T15:28:45Z)
• Neural Strands: Learning Hair Geometry and Appearance from Multi-View Images [40.91569888920849]
  We present Neural Strands, a novel learning framework for modeling accurate hair geometry and appearance from multi-view image inputs. The learned hair model can be rendered in real-time from any viewpoint with high-fidelity view-dependent effects.
  arXiv  Detail & Related papers  (2022-07-28T13:08:46Z)
• I M Avatar: Implicit Morphable Head Avatars from Videos [68.13409777995392]
  We propose IMavatar, a novel method for learning implicit head avatars from monocular videos. Inspired by the fine-grained control mechanisms afforded by conventional 3DMMs, we represent the expression- and pose-related deformations via learned blendshapes and skinning fields. We show quantitatively and qualitatively that our method improves geometry and covers a more complete expression space compared to state-of-the-art methods.
  arXiv  Detail & Related papers  (2021-12-14T15:30:32Z)
• Dynamic Neural Radiance Fields for Monocular 4D Facial Avatar Reconstruction [9.747648609960185]
  We present dynamic neural radiance fields for modeling the appearance and dynamics of a human face. Especially, for telepresence applications in AR or VR, a faithful reproduction of the appearance including novel viewpoints or head-poses is required.
  arXiv  Detail & Related papers  (2020-12-05T16:01:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.