Synergy between 3DMM and 3D Landmarks for Accurate 3D Facial Geometry

• URL: http://arxiv.org/abs/2110.09772v3
• Date: Wed, 17 Jan 2024 07:38:17 GMT
• Title: Synergy between 3DMM and 3D Landmarks for Accurate 3D Facial Geometry
• Authors: Cho-Ying Wu, Qiangeng Xu, Ulrich Neumann
• Abstract summary: This work studies learning from a synergy process of 3D Morphable Models (3DMM) and 3D facial landmarks. We predict complete 3D facial geometry, including 3D alignment, face orientation, and 3D face modeling.
• Score: 21.051258644469268
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This work studies learning from a synergy process of 3D Morphable Models (3DMM) and 3D facial landmarks to predict complete 3D facial geometry, including 3D alignment, face orientation, and 3D face modeling. Our synergy process leverages a representation cycle for 3DMM parameters and 3D landmarks. 3D landmarks can be extracted and refined from face meshes built by 3DMM parameters. We next reverse the representation direction and show that predicting 3DMM parameters from sparse 3D landmarks improves the information flow. Together we create a synergy process that utilizes the relation between 3D landmarks and 3DMM parameters, and they collaboratively contribute to better performance. We extensively validate our contribution on full tasks of facial geometry prediction and show our superior and robust performance on these tasks for various scenarios. Particularly, we adopt only simple and widely-used network operations to attain fast and accurate facial geometry prediction. Codes and data: https://choyingw.github.io/works/SynergyNet/

Related papers

• Deep Geometric Moments Promote Shape Consistency in Text-to-3D Generation [27.43973967994717]
  MT3D is a text-to-3D generative model that leverages a high-fidelity 3D object to overcome viewpoint bias. We employ depth maps derived from a high-quality 3D model as control signals to guarantee that the generated 2D images preserve the fundamental shape and structure. By incorporating geometric details from a 3D asset, MT3D enables the creation of diverse and geometrically consistent objects.
  arXiv  Detail & Related papers  (2024-08-12T06:25:44Z)
• DIRECT-3D: Learning Direct Text-to-3D Generation on Massive Noisy 3D Data [50.164670363633704]
  We present DIRECT-3D, a diffusion-based 3D generative model for creating high-quality 3D assets from text prompts. Our model is directly trained on extensive noisy and unaligned in-the-wild' 3D assets. We achieve state-of-the-art performance in both single-class generation and text-to-3D generation.
  arXiv  Detail & Related papers  (2024-06-06T17:58:15Z)
• DNPM: A Neural Parametric Model for the Synthesis of Facial Geometric Details [8.849223786633083]
  In 3D face modeling, 3DMM is the most widely used parametric model, but can't generate fine geometric details solely from identity and expression inputs. We propose a neural parametric model named DNPM for the facial geometric details, which utilizes deep neural network to extract latent codes from facial displacement maps encoding details and wrinkles. We show that DNPM and Detailed3DMM can facilitate two downstream applications: speech-driven detailed 3D facial animation and 3D face reconstruction from a degraded image.
  arXiv  Detail & Related papers  (2024-05-30T04:57:55Z)
• CC3D: Layout-Conditioned Generation of Compositional 3D Scenes [49.281006972028194]
  We introduce CC3D, a conditional generative model that synthesizes complex 3D scenes conditioned on 2D semantic scene layouts. Our evaluations on synthetic 3D-FRONT and real-world KITTI-360 datasets demonstrate that our model generates scenes of improved visual and geometric quality.
  arXiv  Detail & Related papers  (2023-03-21T17:59:02Z)
• Joint-MAE: 2D-3D Joint Masked Autoencoders for 3D Point Cloud Pre-training [65.75399500494343]
  Masked Autoencoders (MAE) have shown promising performance in self-supervised learning for 2D and 3D computer vision. We propose Joint-MAE, a 2D-3D joint MAE framework for self-supervised 3D point cloud pre-training.
  arXiv  Detail & Related papers  (2023-02-27T17:56:18Z)
• RAFaRe: Learning Robust and Accurate Non-parametric 3D Face Reconstruction from Pseudo 2D&3D Pairs [13.11105614044699]
  We propose a robust and accurate non-parametric method for single-view 3D face reconstruction (SVFR) A large-scale pseudo 2D&3D dataset is created by first rendering the detailed 3D faces, then swapping the face in the wild images with the rendered face. Our model outperforms previous methods on FaceScape-wild/lab and MICC benchmarks.
  arXiv  Detail & Related papers  (2023-02-10T19:40:26Z)
• Beyond 3DMM: Learning to Capture High-fidelity 3D Face Shape [77.95154911528365]
  3D Morphable Model (3DMM) fitting has widely benefited face analysis due to its strong 3D priori. Previous reconstructed 3D faces suffer from degraded visual verisimilitude due to the loss of fine-grained geometry. This paper proposes a complete solution to capture the personalized shape so that the reconstructed shape looks identical to the corresponding person.
  arXiv  Detail & Related papers  (2022-04-09T03:46:18Z)
• Asymmetric 3D Context Fusion for Universal Lesion Detection [55.61873234187917]
  3D networks are strong in 3D context yet lack supervised pretraining. Existing 3D context fusion operators are designed to be spatially symmetric, performing identical operations on each 2D slice like convolutions. We propose a novel asymmetric 3D context fusion operator (A3D), which uses different weights to fuse 3D context from different 2D slices.
  arXiv  Detail & Related papers  (2021-09-17T16:25:10Z)
• Accurate 3D Facial Geometry Prediction by Multi-Task, Multi-Modal, and Multi-Representation Landmark Refinement Network [14.966695101335704]
  This work focuses on complete 3D facial geometry prediction, including 3D facial alignment via 3D modeling and face orientation estimation. Our focus is on the important facial attributes, 3D landmarks, and we fully utilize their embedded information to guide 3D facial geometry learning. We attain the state of the art from extensive experiments on all tasks of learning 3D facial geometry.
  arXiv  Detail & Related papers  (2021-04-16T23:22:41Z)
• Cylinder3D: An Effective 3D Framework for Driving-scene LiDAR Semantic Segmentation [87.54570024320354]
  State-of-the-art methods for large-scale driving-scene LiDAR semantic segmentation often project and process the point clouds in the 2D space. A straightforward solution to tackle the issue of 3D-to-2D projection is to keep the 3D representation and process the points in the 3D space. We develop a 3D cylinder partition and a 3D cylinder convolution based framework, termed as Cylinder3D, which exploits the 3D topology relations and structures of driving-scene point clouds.
  arXiv  Detail & Related papers  (2020-08-04T13:56:19Z)
• FaceScape: a Large-scale High Quality 3D Face Dataset and Detailed Riggable 3D Face Prediction [39.95272819738226]
  We present a novel algorithm that is able to predict elaborate riggable 3D face models from a single image input. FaceScape dataset provides 18,760 textured 3D faces, captured from 938 subjects and each with 20 specific expressions.
  arXiv  Detail & Related papers  (2020-03-31T07:11:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.