JNR: Joint-based Neural Rig Representation for Compact 3D Face Modeling

• URL: http://arxiv.org/abs/2007.06755v3
• Date: Fri, 17 Jul 2020 23:35:13 GMT
• Title: JNR: Joint-based Neural Rig Representation for Compact 3D Face Modeling
• Authors: Noranart Vesdapunt, Mitch Rundle, HsiangTao Wu, Baoyuan Wang
• Abstract summary: We introduce a novel approach to learn a 3D face model using a joint-based face rig and a neural skinning network. Thanks to the joint-based representation, our model enjoys some significant advantages over prior blendshape-based models.
• Score: 22.584569656416864
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In this paper, we introduce a novel approach to learn a 3D face model using a joint-based face rig and a neural skinning network. Thanks to the joint-based representation, our model enjoys some significant advantages over prior blendshape-based models. First, it is very compact such that we are orders of magnitude smaller while still keeping strong modeling capacity. Second, because each joint has its semantic meaning, interactive facial geometry editing is made easier and more intuitive. Third, through skinning, our model supports adding mouth interior and eyes, as well as accessories (hair, eye glasses, etc.) in a simpler, more accurate and principled way. We argue that because the human face is highly structured and topologically consistent, it does not need to be learned entirely from data. Instead we can leverage prior knowledge in the form of a human-designed 3D face rig to reduce the data dependency, and learn a compact yet strong face model from only a small dataset (less than one hundred 3D scans). To further improve the modeling capacity, we train a skinning weight generator through adversarial learning. Experiments on fitting high-quality 3D scans (both neutral and expressive), noisy depth images, and RGB images demonstrate that its modeling capacity is on-par with state-of-the-art face models, such as FLAME and Facewarehouse, even though the model is 10 to 20 times smaller. This suggests broad value in both graphics and vision applications on mobile and edge devices.

Related papers

• SketchMetaFace: A Learning-based Sketching Interface for High-fidelity 3D Character Face Modeling [69.28254439393298]
  SketchMetaFace is a sketching system targeting amateur users to model high-fidelity 3D faces in minutes. We develop a novel learning-based method termed "Implicit and Depth Guided Mesh Modeling" (IDGMM) It fuses the advantages of mesh, implicit, and depth representations to achieve high-quality results with high efficiency.
  arXiv  Detail & Related papers  (2023-07-03T07:41:07Z)
• DANBO: Disentangled Articulated Neural Body Representations via Graph Neural Networks [12.132886846993108]
  High-resolution models enable photo-realistic avatars but at the cost of requiring studio settings not available to end users. Our goal is to create avatars directly from raw images without relying on expensive studio setups and surface tracking. We introduce a three-stage method that induces two inductive biases to better disentangled pose-dependent deformation.
  arXiv  Detail & Related papers  (2022-05-03T17:56:46Z)
• DRaCoN -- Differentiable Rasterization Conditioned Neural Radiance Fields for Articulated Avatars [92.37436369781692]
  We present DRaCoN, a framework for learning full-body volumetric avatars. It exploits the advantages of both the 2D and 3D neural rendering techniques. Experiments on the challenging ZJU-MoCap and Human3.6M datasets indicate that DRaCoN outperforms state-of-the-art methods.
  arXiv  Detail & Related papers  (2022-03-29T17:59:15Z)
• FaceVerse: a Fine-grained and Detail-controllable 3D Face Morphable Model from a Hybrid Dataset [36.688730105295015]
  FaceVerse is built from hybrid East Asian face datasets containing 60K fused RGB-D images and 2K high-fidelity 3D head scan models. In the coarse module, we generate a base parametric model from large-scale RGB-D images, which is able to predict accurate rough 3D face models in different genders, ages, etc. In the fine module, a conditional StyleGAN architecture trained with high-fidelity scan models is introduced to enrich elaborate facial geometric and texture details.
  arXiv  Detail & Related papers  (2022-03-26T12:13:14Z)
• Building 3D Generative Models from Minimal Data [3.472931603805115]
  We show that our approach can be used to perform face recognition using only a single 3D template (one scan total, not one per person) We extend our model to a preliminary unsupervised learning framework that enables the learning of the distribution of 3D faces using one 3D template and a small number of 2D images.
  arXiv  Detail & Related papers  (2022-03-04T20:10:50Z)
• BANMo: Building Animatable 3D Neural Models from Many Casual Videos [135.64291166057373]
  We present BANMo, a method that requires neither a specialized sensor nor a pre-defined template shape. Banmo builds high-fidelity, articulated 3D models from many monocular casual videos in a differentiable rendering framework. On real and synthetic datasets, BANMo shows higher-fidelity 3D reconstructions than prior works for humans and animals.
  arXiv  Detail & Related papers  (2021-12-23T18:30:31Z)
• Monocular Real-time Full Body Capture with Inter-part Correlations [66.22835689189237]
  We present the first method for real-time full body capture that estimates shape and motion of body and hands together with a dynamic 3D face model from a single color image. Our approach uses a new neural network architecture that exploits correlations between body and hands at high computational efficiency.
  arXiv  Detail & Related papers  (2020-12-11T02:37:56Z)
• SMPLpix: Neural Avatars from 3D Human Models [56.85115800735619]
  We bridge the gap between classic rendering and the latest generative networks operating in pixel space. We train a network that directly converts a sparse set of 3D mesh vertices into photorealistic images. We show the advantage over conventional differentiables both in terms of the level of photorealism and rendering efficiency.
  arXiv  Detail & Related papers  (2020-08-16T10:22:00Z)
• Combining Implicit Function Learning and Parametric Models for 3D Human Reconstruction [123.62341095156611]
  Implicit functions represented as deep learning approximations are powerful for reconstructing 3D surfaces. Such features are essential in building flexible models for both computer graphics and computer vision. We present methodology that combines detail-rich implicit functions and parametric representations.
  arXiv  Detail & Related papers  (2020-07-22T13:46:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.