A Large-Scale 3D Face Mesh Video Dataset via Neural Re-parameterized Optimization

• URL: http://arxiv.org/abs/2310.03205v2
• Date: Fri, 6 Oct 2023 14:37:58 GMT
• Title: A Large-Scale 3D Face Mesh Video Dataset via Neural Re-parameterized Optimization
• Authors: Kim Youwang and Lee Hyun and Kim Sung-Bin and Suekyeong Nam and Janghoon Ju and Tae-Hyun Oh
• Abstract summary: We propose NeuFace, a 3D face mesh pseudo annotation method on videos. We annotate the per-view/frame accurate and consistent face meshes on large-scale face videos, called the NeuFace-dataset. By exploiting the naturalness and diversity of 3D faces in our dataset, we demonstrate the usefulness of our dataset for 3D face-related tasks.
• Score: 17.938604013181426
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We propose NeuFace, a 3D face mesh pseudo annotation method on videos via neural re-parameterized optimization. Despite the huge progress in 3D face reconstruction methods, generating reliable 3D face labels for in-the-wild dynamic videos remains challenging. Using NeuFace optimization, we annotate the per-view/-frame accurate and consistent face meshes on large-scale face videos, called the NeuFace-dataset. We investigate how neural re-parameterization helps to reconstruct image-aligned facial details on 3D meshes via gradient analysis. By exploiting the naturalness and diversity of 3D faces in our dataset, we demonstrate the usefulness of our dataset for 3D face-related tasks: improving the reconstruction accuracy of an existing 3D face reconstruction model and learning 3D facial motion prior. Code and datasets will be available at https://neuface-dataset.github.io.

Related papers

• FaceGPT: Self-supervised Learning to Chat about 3D Human Faces [69.4651241319356]
  We introduce FaceGPT, a self-supervised learning framework for Large Vision-Language Models (VLMs) to reason about 3D human faces from images and text. FaceGPT overcomes this limitation by embedding the parameters of a 3D morphable face model (3DMM) into the token space of a VLM. We show that FaceGPT achieves high-quality 3D face reconstructions and retains the ability for general-purpose visual instruction following.
  arXiv  Detail & Related papers  (2024-06-11T11:13:29Z)
• RAFaRe: Learning Robust and Accurate Non-parametric 3D Face Reconstruction from Pseudo 2D&3D Pairs [13.11105614044699]
  We propose a robust and accurate non-parametric method for single-view 3D face reconstruction (SVFR) A large-scale pseudo 2D&3D dataset is created by first rendering the detailed 3D faces, then swapping the face in the wild images with the rendered face. Our model outperforms previous methods on FaceScape-wild/lab and MICC benchmarks.
  arXiv  Detail & Related papers  (2023-02-10T19:40:26Z)
• Generating 2D and 3D Master Faces for Dictionary Attacks with a Network-Assisted Latent Space Evolution [68.8204255655161]
  A master face is a face image that passes face-based identity authentication for a high percentage of the population. We optimize these faces for 2D and 3D face verification models. In 3D, we generate faces using the 2D StyleGAN2 generator and predict a 3D structure using a deep 3D face reconstruction network.
  arXiv  Detail & Related papers  (2022-11-25T09:15:38Z)
• FaceScape: 3D Facial Dataset and Benchmark for Single-View 3D Face Reconstruction [29.920622006999732]
  We present a large-scale detailed 3D face dataset, FaceScape, and the corresponding benchmark to evaluate single-view facial 3D reconstruction. By training on FaceScape data, a novel algorithm is proposed to predict elaborate riggable 3D face models from a single image input. We also use FaceScape data to generate the in-the-wild and in-the-lab benchmark to evaluate recent methods of single-view face reconstruction.
  arXiv  Detail & Related papers  (2021-11-01T16:48:34Z)
• Image-to-Video Generation via 3D Facial Dynamics [78.01476554323179]
  We present a versatile model, FaceAnime, for various video generation tasks from still images. Our model is versatile for various AR/VR and entertainment applications, such as face video and face video prediction.
  arXiv  Detail & Related papers  (2021-05-31T02:30:11Z)
• Fast-GANFIT: Generative Adversarial Network for High Fidelity 3D Face Reconstruction [76.1612334630256]
  We harness the power of Generative Adversarial Networks (GANs) and Deep Convolutional Neural Networks (DCNNs) to reconstruct the facial texture and shape from single images. We demonstrate excellent results in photorealistic and identity preserving 3D face reconstructions and achieve for the first time, facial texture reconstruction with high-frequency details.
  arXiv  Detail & Related papers  (2021-05-16T16:35:44Z)
• Realistic face animation generation from videos [2.398608007786179]
  3D face reconstruction and face alignment are two fundamental and highly related topics in computer vision. Recently, some works start to use deep learning models to estimate the 3DMM coefficients to reconstruct 3D face geometry. To address this problem, some end-to-end methods, which can completely bypass the calculation of 3DMM coefficients, are proposed.
  arXiv  Detail & Related papers  (2021-03-27T20:18:14Z)
• OSTeC: One-Shot Texture Completion [86.23018402732748]
  We propose an unsupervised approach for one-shot 3D facial texture completion. The proposed approach rotates an input image in 3D and fill-in the unseen regions by reconstructing the rotated image in a 2D face generator. We frontalize the target image by projecting the completed texture into the generator.
  arXiv  Detail & Related papers  (2020-12-30T23:53:26Z)
• DeepFaceFlow: In-the-wild Dense 3D Facial Motion Estimation [56.56575063461169]
  DeepFaceFlow is a robust, fast, and highly-accurate framework for the estimation of 3D non-rigid facial flow. Our framework was trained and tested on two very large-scale facial video datasets. Given registered pairs of images, our framework generates 3D flow maps at 60 fps.
  arXiv  Detail & Related papers  (2020-05-14T23:56:48Z)
• FaceScape: a Large-scale High Quality 3D Face Dataset and Detailed Riggable 3D Face Prediction [39.95272819738226]
  We present a novel algorithm that is able to predict elaborate riggable 3D face models from a single image input. FaceScape dataset provides 18,760 textured 3D faces, captured from 938 subjects and each with 20 specific expressions.
  arXiv  Detail & Related papers  (2020-03-31T07:11:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.