Perspective Reconstruction of Human Faces by Joint Mesh and Landmark Regression

• URL: http://arxiv.org/abs/2208.07142v1
• Date: Mon, 15 Aug 2022 12:32:20 GMT
• Title: Perspective Reconstruction of Human Faces by Joint Mesh and Landmark Regression
• Authors: Jia Guo, Jinke Yu, Alexandros Lattas, Jiankang Deng
• Abstract summary: We propose to simultaneously reconstruct 3D face mesh in the world space and predict 2D face landmarks on the image plane. Based on the predicted 3D and 2D landmarks, the 6DoF (6 Degrees Freedom) face pose can be easily estimated by the solver.
• Score: 89.8129467907451
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Even though 3D face reconstruction has achieved impressive progress, most orthogonal projection-based face reconstruction methods can not achieve accurate and consistent reconstruction results when the face is very close to the camera due to the distortion under the perspective projection. In this paper, we propose to simultaneously reconstruct 3D face mesh in the world space and predict 2D face landmarks on the image plane to address the problem of perspective 3D face reconstruction. Based on the predicted 3D vertices and 2D landmarks, the 6DoF (6 Degrees of Freedom) face pose can be easily estimated by the PnP solver to represent perspective projection. Our approach achieves 1st place on the leader-board of the ECCV 2022 WCPA challenge and our model is visually robust under different identities, expressions and poses. The training code and models are released to facilitate future research.

Related papers

• A Hierarchical Representation Network for Accurate and Detailed Face Reconstruction from In-The-Wild Images [15.40230841242637]
  We present a novel hierarchical representation network (HRN) to achieve accurate and detailed face reconstruction from a single image. Our framework can be extended to a multi-view fashion by considering detail consistency of different views. Our method outperforms the existing methods in both reconstruction accuracy and visual effects.
  arXiv  Detail & Related papers  (2023-02-28T09:24:36Z)
• Towards 3D Face Reconstruction in Perspective Projection: Estimating 6DoF Face Pose from Monocular Image [48.77844225075744]
  In some scenarios that the face is very close to camera or moving along the camera axis, the methods suffer from the inaccurate reconstruction and unstable temporal fitting. Deep neural network, Perspective Network (PerspNet), is proposed to simultaneously reconstruct 3D face shape in canonical space. We contribute a large ARKitFace dataset to enable the training and evaluation of 3D face reconstruction solutions under the scenarios of perspective projection.
  arXiv  Detail & Related papers  (2022-05-09T08:49:41Z)
• Beyond 3DMM: Learning to Capture High-fidelity 3D Face Shape [77.95154911528365]
  3D Morphable Model (3DMM) fitting has widely benefited face analysis due to its strong 3D priori. Previous reconstructed 3D faces suffer from degraded visual verisimilitude due to the loss of fine-grained geometry. This paper proposes a complete solution to capture the personalized shape so that the reconstructed shape looks identical to the corresponding person.
  arXiv  Detail & Related papers  (2022-04-09T03:46:18Z)
• Realistic face animation generation from videos [2.398608007786179]
  3D face reconstruction and face alignment are two fundamental and highly related topics in computer vision. Recently, some works start to use deep learning models to estimate the 3DMM coefficients to reconstruct 3D face geometry. To address this problem, some end-to-end methods, which can completely bypass the calculation of 3DMM coefficients, are proposed.
  arXiv  Detail & Related papers  (2021-03-27T20:18:14Z)
• Learning 3D Face Reconstruction with a Pose Guidance Network [49.13404714366933]
  We present a self-supervised learning approach to learning monocular 3D face reconstruction with a pose guidance network (PGN) First, we unveil the bottleneck of pose estimation in prior parametric 3D face learning methods, and propose to utilize 3D face landmarks for estimating pose parameters. With our specially designed PGN, our model can learn from both faces with fully labeled 3D landmarks and unlimited unlabeled in-the-wild face images.
  arXiv  Detail & Related papers  (2020-10-09T06:11:17Z)
• Adaptive 3D Face Reconstruction from a Single Image [45.736818498242016]
  We propose a novel joint 2D and 3D optimization method to adaptively reconstruct 3D face shapes from a single image. Experimental results on multiple datasets demonstrate that our method can generate high-quality reconstruction from a single color image.
  arXiv  Detail & Related papers  (2020-07-08T09:35:26Z)
• AvatarMe: Realistically Renderable 3D Facial Reconstruction "in-the-wild" [105.28776215113352]
  AvatarMe is the first method that is able to reconstruct photorealistic 3D faces from a single "in-the-wild" image with an increasing level of detail. It outperforms the existing arts by a significant margin and reconstructs authentic, 4K by 6K-resolution 3D faces from a single low-resolution image.
  arXiv  Detail & Related papers  (2020-03-30T22:17:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.