Thinking the Fusion Strategy of Multi-reference Face Reenactment

• URL: http://arxiv.org/abs/2202.10758v1
• Date: Tue, 22 Feb 2022 09:17:26 GMT
• Title: Thinking the Fusion Strategy of Multi-reference Face Reenactment
• Authors: Takuya Yashima, Takuya Narihira, Tamaki Kojima
• Abstract summary: We show that simple extension by using multiple reference images significantly improves generation quality. We show this by 1) conducting the reconstruction task on publicly available dataset, 2) conducting facial motion transfer on our original dataset which consists of multi-person's head movement video sequences, and 3) using a newly proposed evaluation metric to validate that our method achieves better quantitative results.
• Score: 4.1509697008011175
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In recent advances of deep generative models, face reenactment -manipulating and controlling human face, including their head movement-has drawn much attention for its wide range of applicability. Despite its strong expressiveness, it is inevitable that the models fail to reconstruct or accurately generate unseen side of the face of a given single reference image. Most of existing methods alleviate this problem by learning appearances of human faces from large amount of data and generate realistic texture at inference time. Rather than completely relying on what generative models learn, we show that simple extension by using multiple reference images significantly improves generation quality. We show this by 1) conducting the reconstruction task on publicly available dataset, 2) conducting facial motion transfer on our original dataset which consists of multi-person's head movement video sequences, and 3) using a newly proposed evaluation metric to validate that our method achieves better quantitative results.

Related papers

• Single Image, Any Face: Generalisable 3D Face Generation [59.9369171926757]
  We propose a novel model, Gen3D-Face, which generates 3D human faces with unconstrained single image input. To the best of our knowledge, this is the first attempt and benchmark for creating photorealistic 3D human face avatars from single images.
  arXiv  Detail & Related papers  (2024-09-25T14:56:37Z)
• SPARK: Self-supervised Personalized Real-time Monocular Face Capture [6.093606972415841]
  Current state of the art approaches have the ability to regress parametric 3D face models in real-time across a wide range of identities. We propose a method for high-precision 3D face capture taking advantage of a collection of unconstrained videos of a subject as prior information.
  arXiv  Detail & Related papers  (2024-09-12T12:30:04Z)
• Effective Adapter for Face Recognition in the Wild [72.75516495170199]
  We tackle the challenge of face recognition in the wild, where images often suffer from low quality and real-world distortions. Traditional approaches-either training models directly on degraded images or their enhanced counterparts using face restoration techniques-have proven ineffective. We propose an effective adapter for augmenting existing face recognition models trained on high-quality facial datasets.
  arXiv  Detail & Related papers  (2023-12-04T08:55:46Z)
• HyperReenact: One-Shot Reenactment via Jointly Learning to Refine and Retarget Faces [47.27033282706179]
  We present our method for neural face reenactment, called HyperReenact, that aims to generate realistic talking head images of a source identity. Our method operates under the one-shot setting (i.e., using a single source frame) and allows for cross-subject reenactment, without requiring subject-specific fine-tuning. We compare our method both quantitatively and qualitatively against several state-of-the-art techniques on the standard benchmarks of VoxCeleb1 and VoxCeleb2.
  arXiv  Detail & Related papers  (2023-07-20T11:59:42Z)
• A Hierarchical Representation Network for Accurate and Detailed Face Reconstruction from In-The-Wild Images [15.40230841242637]
  We present a novel hierarchical representation network (HRN) to achieve accurate and detailed face reconstruction from a single image. Our framework can be extended to a multi-view fashion by considering detail consistency of different views. Our method outperforms the existing methods in both reconstruction accuracy and visual effects.
  arXiv  Detail & Related papers  (2023-02-28T09:24:36Z)
• Semantic-aware One-shot Face Re-enactment with Dense Correspondence Estimation [100.60938767993088]
  One-shot face re-enactment is a challenging task due to the identity mismatch between source and driving faces. This paper proposes to use 3D Morphable Model (3DMM) for explicit facial semantic decomposition and identity disentanglement.
  arXiv  Detail & Related papers  (2022-11-23T03:02:34Z)
• Everything's Talkin': Pareidolia Face Reenactment [119.49707201178633]
  Pareidolia Face Reenactment is defined as animating a static illusory face to move in tandem with a human face in the video. For the large differences between pareidolia face reenactment and traditional human face reenactment, shape variance and texture variance are introduced. We propose a novel Parametric Unsupervised Reenactment Algorithm to tackle these two challenges.
  arXiv  Detail & Related papers  (2021-04-07T11:19:13Z)
• Weakly-Supervised Multi-Face 3D Reconstruction [45.864415499303405]
  We propose an effective end-to-end framework for multi-face 3D reconstruction. We employ the same global camera model for the reconstructed faces in each image, which makes it possible to recover the relative head positions and orientations in the 3D scene.
  arXiv  Detail & Related papers  (2021-01-06T13:15:21Z)
• Learning Complete 3D Morphable Face Models from Images and Videos [88.34033810328201]
  We present the first approach to learn complete 3D models of face identity geometry, albedo and expression just from images and videos. We show that our learned models better generalize and lead to higher quality image-based reconstructions than existing approaches.
  arXiv  Detail & Related papers  (2020-10-04T20:51:23Z)
• Rotate-and-Render: Unsupervised Photorealistic Face Rotation from Single-View Images [47.18219551855583]
  We propose a novel unsupervised framework that can synthesize photo-realistic rotated faces. Our key insight is that rotating faces in the 3D space back and forth, and re-rendering them to the 2D plane can serve as a strong self-supervision. Our approach has superior synthesis quality as well as identity preservation over the state-of-the-art methods.
  arXiv  Detail & Related papers  (2020-03-18T09:54:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.