Finding Directions in GAN's Latent Space for Neural Face Reenactment

• URL: http://arxiv.org/abs/2202.00046v1
• Date: Mon, 31 Jan 2022 19:14:03 GMT
• Title: Finding Directions in GAN's Latent Space for Neural Face Reenactment
• Authors: Stella Bounareli, Vasileios Argyriou, Georgios Tzimiropoulos
• Abstract summary: This paper is on face/head reenactment where the goal is to transfer the facial pose (3D head orientation and expression) of a target face to a source face. We take a different approach, bypassing the training of such networks, by using (fine-tuned) pre-trained GANs. We show that by embedding real images in the GAN latent space, our method can be successfully used for the reenactment of real-world faces.
• Score: 45.67273942952348
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This paper is on face/head reenactment where the goal is to transfer the facial pose (3D head orientation and expression) of a target face to a source face. Previous methods focus on learning embedding networks for identity and pose disentanglement which proves to be a rather hard task, degrading the quality of the generated images. We take a different approach, bypassing the training of such networks, by using (fine-tuned) pre-trained GANs which have been shown capable of producing high-quality facial images. Because GANs are characterized by weak controllability, the core of our approach is a method to discover which directions in latent GAN space are responsible for controlling facial pose and expression variations. We present a simple pipeline to learn such directions with the aid of a 3D shape model which, by construction, already captures disentangled directions for facial pose, identity and expression. Moreover, we show that by embedding real images in the GAN latent space, our method can be successfully used for the reenactment of real-world faces. Our method features several favorable properties including using a single source image (one-shot) and enabling cross-person reenactment. Our qualitative and quantitative results show that our approach often produces reenacted faces of significantly higher quality than those produced by state-of-the-art methods for the standard benchmarks of VoxCeleb1 & 2.

Related papers

• One-shot Neural Face Reenactment via Finding Directions in GAN's Latent Space [37.357842761713705]
  We present a framework for neural face/head reenactment whose goal is to transfer the 3D head orientation and expression of a target face to a source face. Our method features several favorable properties including using a single source image (one-shot) and enabling cross-person reenactment.
  arXiv  Detail & Related papers  (2024-02-05T22:12:42Z)
• HyperReenact: One-Shot Reenactment via Jointly Learning to Refine and Retarget Faces [47.27033282706179]
  We present our method for neural face reenactment, called HyperReenact, that aims to generate realistic talking head images of a source identity. Our method operates under the one-shot setting (i.e., using a single source frame) and allows for cross-subject reenactment, without requiring subject-specific fine-tuning. We compare our method both quantitatively and qualitatively against several state-of-the-art techniques on the standard benchmarks of VoxCeleb1 and VoxCeleb2.
  arXiv  Detail & Related papers  (2023-07-20T11:59:42Z)
• Semantic-aware One-shot Face Re-enactment with Dense Correspondence Estimation [100.60938767993088]
  One-shot face re-enactment is a challenging task due to the identity mismatch between source and driving faces. This paper proposes to use 3D Morphable Model (3DMM) for explicit facial semantic decomposition and identity disentanglement.
  arXiv  Detail & Related papers  (2022-11-23T03:02:34Z)
• Graph-based Generative Face Anonymisation with Pose Preservation [49.18049578591058]
  AnonyGAN is a GAN-based solution for face anonymisation. It replaces the visual information corresponding to a source identity with a condition identity provided as any single image.
  arXiv  Detail & Related papers  (2021-12-10T12:58:17Z)
• MOST-GAN: 3D Morphable StyleGAN for Disentangled Face Image Manipulation [69.35523133292389]
  We propose a framework that a priori models physical attributes of the face explicitly, thus providing disentanglement by design. Our method, MOST-GAN, integrates the expressive power and photorealism of style-based GANs with the physical disentanglement and flexibility of nonlinear 3D morphable models. It achieves photorealistic manipulation of portrait images with fully disentangled 3D control over their physical attributes, enabling extreme manipulation of lighting, facial expression, and pose variations up to full profile view.
  arXiv  Detail & Related papers  (2021-11-01T15:53:36Z)
• Learning to Aggregate and Personalize 3D Face from In-the-Wild Photo Collection [65.92058628082322]
  Non-parametric face modeling aims to reconstruct 3D face only from images without shape assumptions. This paper presents a novel Learning to Aggregate and Personalize framework for unsupervised robust 3D face modeling.
  arXiv  Detail & Related papers  (2021-06-15T03:10:17Z)
• Mesh Guided One-shot Face Reenactment using Graph Convolutional Networks [31.083072922977568]
  We introduce a method for one-shot face reenactment, which uses the reconstructed 3D meshes as guidance to learn the optical flow needed for the reenacted face synthesis. We propose a motion net to learn the face motion, which is an asymmetric autoencoder. Our method can generate high-quality results and outperforms state-of-the-art methods in both qualitative and quantitative comparisons.
  arXiv  Detail & Related papers  (2020-08-18T07:41:40Z)
• Head2Head++: Deep Facial Attributes Re-Targeting [6.230979482947681]
  We leverage the 3D geometry of faces and Generative Adversarial Networks (GANs) to design a novel deep learning architecture for the task of facial and head reenactment. We manage to capture the complex non-rigid facial motion from the driving monocular performances and synthesise temporally consistent videos. Our system performs end-to-end reenactment in nearly real-time speed (18 fps)
  arXiv  Detail & Related papers  (2020-06-17T23:38:37Z)
• FaR-GAN for One-Shot Face Reenactment [20.894596219099164]
  We present a one-shot face reenactment model, FaR-GAN, that takes only one face image of any given source identity and a target expression as input. The proposed method makes no assumptions about the source identity, facial expression, head pose, or even image background.
  arXiv  Detail & Related papers  (2020-05-13T16:15:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.