Riggable 3D Face Reconstruction via In-Network Optimization

• URL: http://arxiv.org/abs/2104.03493v1
• Date: Thu, 8 Apr 2021 03:53:20 GMT
• Title: Riggable 3D Face Reconstruction via In-Network Optimization
• Authors: Ziqian Bai, Zhaopeng Cui, Xiaoming Liu, Ping Tan
• Abstract summary: This paper presents a method for riggable 3D face reconstruction from monocular images. It jointly estimates a personalized face rig and per-image parameters including expressions, poses, and illuminations. Experiments demonstrate that our method achieves SOTA reconstruction accuracy, reasonable robustness and generalization ability.
• Score: 58.016067611038046
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper presents a method for riggable 3D face reconstruction from monocular images, which jointly estimates a personalized face rig and per-image parameters including expressions, poses, and illuminations. To achieve this goal, we design an end-to-end trainable network embedded with a differentiable in-network optimization. The network first parameterizes the face rig as a compact latent code with a neural decoder, and then estimates the latent code as well as per-image parameters via a learnable optimization. By estimating a personalized face rig, our method goes beyond static reconstructions and enables downstream applications such as video retargeting. In-network optimization explicitly enforces constraints derived from the first principles, thus introduces additional priors than regression-based methods. Finally, data-driven priors from deep learning are utilized to constrain the ill-posed monocular setting and ease the optimization difficulty. Experiments demonstrate that our method achieves SOTA reconstruction accuracy, reasonable robustness and generalization ability, and supports standard face rig applications.

Related papers

• Hyper-VolTran: Fast and Generalizable One-Shot Image to 3D Object Structure via HyperNetworks [53.67497327319569]
  We introduce a novel neural rendering technique to solve image-to-3D from a single view. Our approach employs the signed distance function as the surface representation and incorporates generalizable priors through geometry-encoding volumes and HyperNetworks. Our experiments show the advantages of our proposed approach with consistent results and rapid generation.
  arXiv  Detail & Related papers  (2023-12-24T08:42:37Z)
• A Probabilistic Deep Image Prior for Computational Tomography [0.19573380763700707]
  Existing deep-learning based tomographic image reconstruction methods do not provide accurate estimates of reconstruction uncertainty. We construct a Bayesian prior for tomographic reconstruction, which combines the classical total variation (TV) regulariser with the modern deep image prior (DIP) For the inference, we develop an approach based on the linearised Laplace method, which is scalable to high-dimensional settings.
  arXiv  Detail & Related papers  (2022-02-28T14:47:14Z)
• NerfingMVS: Guided Optimization of Neural Radiance Fields for Indoor Multi-view Stereo [97.07453889070574]
  We present a new multi-view depth estimation method that utilizes both conventional SfM reconstruction and learning-based priors. We show that our proposed framework significantly outperforms state-of-the-art methods on indoor scenes.
  arXiv  Detail & Related papers  (2021-09-02T17:54:31Z)
• SIDER: Single-Image Neural Optimization for Facial Geometric Detail Recovery [54.64663713249079]
  SIDER is a novel photometric optimization method that recovers detailed facial geometry from a single image in an unsupervised manner. In contrast to prior work, SIDER does not rely on any dataset priors and does not require additional supervision from multiple views, lighting changes or ground truth 3D shape.
  arXiv  Detail & Related papers  (2021-08-11T22:34:53Z)
• Fast-GANFIT: Generative Adversarial Network for High Fidelity 3D Face Reconstruction [76.1612334630256]
  We harness the power of Generative Adversarial Networks (GANs) and Deep Convolutional Neural Networks (DCNNs) to reconstruct the facial texture and shape from single images. We demonstrate excellent results in photorealistic and identity preserving 3D face reconstructions and achieve for the first time, facial texture reconstruction with high-frequency details.
  arXiv  Detail & Related papers  (2021-05-16T16:35:44Z)
• Inverting Generative Adversarial Renderer for Face Reconstruction [58.45125455811038]
  In this work, we introduce a novel Generative Adversa Renderer (GAR) GAR learns to model the complicated real-world image, instead of relying on the graphics rules, it is capable of producing realistic images. Our method achieves state-of-the-art performances on multiple face reconstruction.
  arXiv  Detail & Related papers  (2021-05-06T04:16:06Z)
• Weakly-Supervised Multi-Face 3D Reconstruction [45.864415499303405]
  We propose an effective end-to-end framework for multi-face 3D reconstruction. We employ the same global camera model for the reconstructed faces in each image, which makes it possible to recover the relative head positions and orientations in the 3D scene.
  arXiv  Detail & Related papers  (2021-01-06T13:15:21Z)
• Deep Probabilistic Feature-metric Tracking [27.137827823264942]
  We propose a new framework to learn a pixel-wise deep feature map and a deep feature-metric uncertainty map. CNN predicts a deep initial pose for faster and more reliable convergence. Experimental results demonstrate state-of-the-art performances on the TUM RGB-D dataset and the 3D rigid object tracking dataset.
  arXiv  Detail & Related papers  (2020-08-31T11:47:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.