VRMM: A Volumetric Relightable Morphable Head Model

• URL: http://arxiv.org/abs/2402.04101v2
• Date: Wed, 8 May 2024 15:45:34 GMT
• Title: VRMM: A Volumetric Relightable Morphable Head Model
• Authors: Haotian Yang, Mingwu Zheng, Chongyang Ma, Yu-Kun Lai, Pengfei Wan, Haibin Huang,
• Abstract summary: We introduce the Volumetric Relightable Morphable Model (VRMM), a novel volumetric and parametric facial prior for 3D face modeling. Our framework efficiently disentangles and encodes latent spaces of identity, expression, and lighting into low-dimensional representations. We demonstrate the versatility and effectiveness of VRMM through various applications like avatar generation, facial reconstruction, and animation.
• Score: 55.21098471673929
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In this paper, we introduce the Volumetric Relightable Morphable Model (VRMM), a novel volumetric and parametric facial prior for 3D face modeling. While recent volumetric prior models offer improvements over traditional methods like 3D Morphable Models (3DMMs), they face challenges in model learning and personalized reconstructions. Our VRMM overcomes these by employing a novel training framework that efficiently disentangles and encodes latent spaces of identity, expression, and lighting into low-dimensional representations. This framework, designed with self-supervised learning, significantly reduces the constraints for training data, making it more feasible in practice. The learned VRMM offers relighting capabilities and encompasses a comprehensive range of expressions. We demonstrate the versatility and effectiveness of VRMM through various applications like avatar generation, facial reconstruction, and animation. Additionally, we address the common issue of overfitting in generative volumetric models with a novel prior-preserving personalization framework based on VRMM. Such an approach enables high-quality 3D face reconstruction from even a single portrait input. Our experiments showcase the potential of VRMM to significantly enhance the field of 3D face modeling.

Related papers

• GPHM: Gaussian Parametric Head Model for Monocular Head Avatar Reconstruction [47.113910048252805]
  High-fidelity 3D human head avatars are crucial for applications in VR/AR, digital human, and film production. Recent advances have leveraged morphable face models to generate animated head avatars, representing varying identities and expressions. We introduce 3D Gaussian Parametric Head Model, which employs 3D Gaussians to accurately represent the complexities of the human head.
  arXiv  Detail & Related papers  (2024-07-21T06:03:11Z)
• FaceFolds: Meshed Radiance Manifolds for Efficient Volumetric Rendering of Dynamic Faces [21.946327323788275]
  3D rendering of dynamic face is a challenging problem. We present a novel representation that enables high-quality rendering of an actor's dynamic facial performances.
  arXiv  Detail & Related papers  (2024-04-22T00:44:13Z)
• InstantMesh: Efficient 3D Mesh Generation from a Single Image with Sparse-view Large Reconstruction Models [66.83681825842135]
  InstantMesh is a feed-forward framework for instant 3D mesh generation from a single image. It features state-of-the-art generation quality and significant training scalability. We release all the code, weights, and demo of InstantMesh with the intention that it can make substantial contributions to the community of 3D generative AI.
  arXiv  Detail & Related papers  (2024-04-10T17:48:37Z)
• SketchMetaFace: A Learning-based Sketching Interface for High-fidelity 3D Character Face Modeling [69.28254439393298]
  SketchMetaFace is a sketching system targeting amateur users to model high-fidelity 3D faces in minutes. We develop a novel learning-based method termed "Implicit and Depth Guided Mesh Modeling" (IDGMM) It fuses the advantages of mesh, implicit, and depth representations to achieve high-quality results with high efficiency.
  arXiv  Detail & Related papers  (2023-07-03T07:41:07Z)
• MA-NeRF: Motion-Assisted Neural Radiance Fields for Face Synthesis from Sparse Images [21.811067296567252]
  We propose a novel framework that can reconstruct a high-fidelity drivable face avatar and handle unseen expressions. At the core of our implementation are structured displacement feature and semantic-aware learning module. Our method achieves much better results than the current state-of-the-arts.
  arXiv  Detail & Related papers  (2023-06-17T13:49:56Z)
• ASM: Adaptive Skinning Model for High-Quality 3D Face Modeling [11.885382595302751]
  We argue that reconstruction with multi-view uncalibrated images demands a new model with stronger capacity. We propose Adaptive Skinning Model (ASM), which redefines the skinning model with more compact and fully tunable parameters. Our work opens up new research direction for parametric face model and facilitates future research on multi-view reconstruction.
  arXiv  Detail & Related papers  (2023-04-19T04:55:28Z)
• Learning Personalized High Quality Volumetric Head Avatars from Monocular RGB Videos [47.94545609011594]
  We propose a method to learn a high-quality implicit 3D head avatar from a monocular RGB video captured in the wild. Our hybrid pipeline combines the geometry prior and dynamic tracking of a 3DMM with a neural radiance field to achieve fine-grained control and photorealism.
  arXiv  Detail & Related papers  (2023-04-04T01:10:04Z)
• FNeVR: Neural Volume Rendering for Face Animation [53.92664037596834]
  We propose a Face Neural Volume Rendering (FNeVR) network to explore the potential of 2D motion warping and 3D volume rendering. In FNeVR, we design a 3D Face Volume Rendering (FVR) module to enhance the facial details for image rendering. We also design a lightweight pose editor, enabling FNeVR to edit the facial pose in a simple yet effective way.
  arXiv  Detail & Related papers  (2022-09-21T13:18:59Z)
• Shape My Face: Registering 3D Face Scans by Surface-to-Surface Translation [75.59415852802958]
  Shape-My-Face (SMF) is a powerful encoder-decoder architecture based on an improved point cloud encoder, a novel visual attention mechanism, graph convolutional decoders with skip connections, and a specialized mouth model. Our model provides topologically-sound meshes with minimal supervision, offers faster training time, has orders of magnitude fewer trainable parameters, is more robust to noise, and can generalize to previously unseen datasets.
  arXiv  Detail & Related papers  (2020-12-16T20:02:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.