DNPM: A Neural Parametric Model for the Synthesis of Facial Geometric Details

• URL: http://arxiv.org/abs/2405.19688v2
• Date: Fri, 14 Jun 2024 03:18:50 GMT
• Title: DNPM: A Neural Parametric Model for the Synthesis of Facial Geometric Details
• Authors: Haitao Cao, Baoping Cheng, Qiran Pu, Haocheng Zhang, Bin Luo, Yixiang Zhuang, Juncong Lin, Liyan Chen, Xuan Cheng,
• Abstract summary: In 3D face modeling, 3DMM is the most widely used parametric model, but can't generate fine geometric details solely from identity and expression inputs. We propose a neural parametric model named DNPM for the facial geometric details, which utilizes deep neural network to extract latent codes from facial displacement maps encoding details and wrinkles. We show that DNPM and Detailed3DMM can facilitate two downstream applications: speech-driven detailed 3D facial animation and 3D face reconstruction from a degraded image.
• Score: 8.849223786633083
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Parametric 3D models have enabled a wide variety of computer vision and graphics tasks, such as modeling human faces, bodies and hands. In 3D face modeling, 3DMM is the most widely used parametric model, but can't generate fine geometric details solely from identity and expression inputs. To tackle this limitation, we propose a neural parametric model named DNPM for the facial geometric details, which utilizes deep neural network to extract latent codes from facial displacement maps encoding details and wrinkles. Built upon DNPM, a novel 3DMM named Detailed3DMM is proposed, which augments traditional 3DMMs by including the synthesis of facial details only from the identity and expression inputs. Moreover, we show that DNPM and Detailed3DMM can facilitate two downstream applications: speech-driven detailed 3D facial animation and 3D face reconstruction from a degraded image. Extensive experiments have shown the usefulness of DNPM and Detailed3DMM, and the progressiveness of two proposed applications.

Related papers

• ID-to-3D: Expressive ID-guided 3D Heads via Score Distillation Sampling [96.87575334960258]
  ID-to-3D is a method to generate identity- and text-guided 3D human heads with disentangled expressions. Results achieve an unprecedented level of identity-consistent and high-quality texture and geometry generation.
  arXiv  Detail & Related papers  (2024-05-26T13:36:45Z)
• Learning Personalized High Quality Volumetric Head Avatars from Monocular RGB Videos [47.94545609011594]
  We propose a method to learn a high-quality implicit 3D head avatar from a monocular RGB video captured in the wild. Our hybrid pipeline combines the geometry prior and dynamic tracking of a 3DMM with a neural radiance field to achieve fine-grained control and photorealism.
  arXiv  Detail & Related papers  (2023-04-04T01:10:04Z)
• Next3D: Generative Neural Texture Rasterization for 3D-Aware Head Avatars [36.4402388864691]
  3D-aware generative adversarial networks (GANs) synthesize high-fidelity and multi-view-consistent facial images using only collections of single-view 2D imagery. Recent efforts incorporate 3D Morphable Face Model (3DMM) to describe deformation in generative radiance fields either explicitly or implicitly. We propose a novel 3D GAN framework for unsupervised learning of generative, high-quality and 3D-consistent facial avatars from unstructured 2D images.
  arXiv  Detail & Related papers  (2022-11-21T06:40:46Z)
• Deep-MDS Framework for Recovering the 3D Shape of 2D Landmarks from a Single Image [8.368476827165114]
  This paper proposes a framework to recover the 3D shape of 2D landmarks on a human face, in a single input image. A deep neural network learns the pairwise dissimilarity among 2D landmarks, used by NMDS approach.
  arXiv  Detail & Related papers  (2022-10-27T06:20:10Z)
• Controllable 3D Generative Adversarial Face Model via Disentangling Shape and Appearance [63.13801759915835]
  3D face modeling has been an active area of research in computer vision and computer graphics. This paper proposes a new 3D face generative model that can decouple identity and expression.
  arXiv  Detail & Related papers  (2022-08-30T13:40:48Z)
• Beyond 3DMM: Learning to Capture High-fidelity 3D Face Shape [77.95154911528365]
  3D Morphable Model (3DMM) fitting has widely benefited face analysis due to its strong 3D priori. Previous reconstructed 3D faces suffer from degraded visual verisimilitude due to the loss of fine-grained geometry. This paper proposes a complete solution to capture the personalized shape so that the reconstructed shape looks identical to the corresponding person.
  arXiv  Detail & Related papers  (2022-04-09T03:46:18Z)
• DRaCoN -- Differentiable Rasterization Conditioned Neural Radiance Fields for Articulated Avatars [92.37436369781692]
  We present DRaCoN, a framework for learning full-body volumetric avatars. It exploits the advantages of both the 2D and 3D neural rendering techniques. Experiments on the challenging ZJU-MoCap and Human3.6M datasets indicate that DRaCoN outperforms state-of-the-art methods.
  arXiv  Detail & Related papers  (2022-03-29T17:59:15Z)
• Facial Geometric Detail Recovery via Implicit Representation [147.07961322377685]
  We present a robust texture-guided geometric detail recovery approach using only a single in-the-wild facial image. Our method combines high-quality texture completion with the powerful expressiveness of implicit surfaces. Our method not only recovers accurate facial details but also decomposes normals, albedos, and shading parts in a self-supervised way.
  arXiv  Detail & Related papers  (2022-03-18T01:42:59Z)
• Synergy between 3DMM and 3D Landmarks for Accurate 3D Facial Geometry [21.051258644469268]
  This work studies learning from a synergy process of 3D Morphable Models (3DMM) and 3D facial landmarks. We predict complete 3D facial geometry, including 3D alignment, face orientation, and 3D face modeling.
  arXiv  Detail & Related papers  (2021-10-19T07:29:14Z)
• NPMs: Neural Parametric Models for 3D Deformable Shapes [26.87200488085741]
  We propose a novel, learned alternative to traditional, parametric 3D models. In particular, we learn to disentangle 4D dynamics into latent-space representations of shape and pose. We show that NPMs improve notably over both parametric and non-parametric state of the art in reconstruction and tracking of monocular depth sequences of humans and hands.
  arXiv  Detail & Related papers  (2021-04-01T18:14:56Z)
• Realistic face animation generation from videos [2.398608007786179]
  3D face reconstruction and face alignment are two fundamental and highly related topics in computer vision. Recently, some works start to use deep learning models to estimate the 3DMM coefficients to reconstruct 3D face geometry. To address this problem, some end-to-end methods, which can completely bypass the calculation of 3DMM coefficients, are proposed.
  arXiv  Detail & Related papers  (2021-03-27T20:18:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.