One-shot Neural Face Reenactment via Finding Directions in GAN's Latent Space

• URL: http://arxiv.org/abs/2402.03553v1
• Date: Mon, 5 Feb 2024 22:12:42 GMT
• Title: One-shot Neural Face Reenactment via Finding Directions in GAN's Latent Space
• Authors: Stella Bounareli, Christos Tzelepis, Vasileios Argyriou, Ioannis Patras, Georgios Tzimiropoulos
• Abstract summary: We present a framework for neural face/head reenactment whose goal is to transfer the 3D head orientation and expression of a target face to a source face. Our method features several favorable properties including using a single source image (one-shot) and enabling cross-person reenactment.
• Score: 37.357842761713705
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: In this paper, we present our framework for neural face/head reenactment whose goal is to transfer the 3D head orientation and expression of a target face to a source face. Previous methods focus on learning embedding networks for identity and head pose/expression disentanglement which proves to be a rather hard task, degrading the quality of the generated images. We take a different approach, bypassing the training of such networks, by using (fine-tuned) pre-trained GANs which have been shown capable of producing high-quality facial images. Because GANs are characterized by weak controllability, the core of our approach is a method to discover which directions in latent GAN space are responsible for controlling head pose and expression variations. We present a simple pipeline to learn such directions with the aid of a 3D shape model which, by construction, inherently captures disentangled directions for head pose, identity, and expression. Moreover, we show that by embedding real images in the GAN latent space, our method can be successfully used for the reenactment of real-world faces. Our method features several favorable properties including using a single source image (one-shot) and enabling cross-person reenactment. Extensive qualitative and quantitative results show that our approach typically produces reenacted faces of notably higher quality than those produced by state-of-the-art methods for the standard benchmarks of VoxCeleb1 & 2.

Related papers

• HyperReenact: One-Shot Reenactment via Jointly Learning to Refine and Retarget Faces [47.27033282706179]
  We present our method for neural face reenactment, called HyperReenact, that aims to generate realistic talking head images of a source identity. Our method operates under the one-shot setting (i.e., using a single source frame) and allows for cross-subject reenactment, without requiring subject-specific fine-tuning. We compare our method both quantitatively and qualitatively against several state-of-the-art techniques on the standard benchmarks of VoxCeleb1 and VoxCeleb2.
  arXiv  Detail & Related papers  (2023-07-20T11:59:42Z)
• Neural Point-based Volumetric Avatar: Surface-guided Neural Points for Efficient and Photorealistic Volumetric Head Avatar [62.87222308616711]
  We propose fullname (name), a method that adopts the neural point representation and the neural volume rendering process. Specifically, the neural points are strategically constrained around the surface of the target expression via a high-resolution UV displacement map. By design, our name is better equipped to handle topologically changing regions and thin structures while also ensuring accurate expression control when animating avatars.
  arXiv  Detail & Related papers  (2023-07-11T03:40:10Z)
• Semantic-aware One-shot Face Re-enactment with Dense Correspondence Estimation [100.60938767993088]
  One-shot face re-enactment is a challenging task due to the identity mismatch between source and driving faces. This paper proposes to use 3D Morphable Model (3DMM) for explicit facial semantic decomposition and identity disentanglement.
  arXiv  Detail & Related papers  (2022-11-23T03:02:34Z)
• Finding Directions in GAN's Latent Space for Neural Face Reenactment [45.67273942952348]
  This paper is on face/head reenactment where the goal is to transfer the facial pose (3D head orientation and expression) of a target face to a source face. We take a different approach, bypassing the training of such networks, by using (fine-tuned) pre-trained GANs. We show that by embedding real images in the GAN latent space, our method can be successfully used for the reenactment of real-world faces.
  arXiv  Detail & Related papers  (2022-01-31T19:14:03Z)
• MOST-GAN: 3D Morphable StyleGAN for Disentangled Face Image Manipulation [69.35523133292389]
  We propose a framework that a priori models physical attributes of the face explicitly, thus providing disentanglement by design. Our method, MOST-GAN, integrates the expressive power and photorealism of style-based GANs with the physical disentanglement and flexibility of nonlinear 3D morphable models. It achieves photorealistic manipulation of portrait images with fully disentangled 3D control over their physical attributes, enabling extreme manipulation of lighting, facial expression, and pose variations up to full profile view.
  arXiv  Detail & Related papers  (2021-11-01T15:53:36Z)
• One-shot Face Reenactment Using Appearance Adaptive Normalization [30.615671641713945]
  The paper proposes a novel generative adversarial network for one-shot face reenactment. It can animate a single face image to a different pose-and-expression while keeping its original appearance.
  arXiv  Detail & Related papers  (2021-02-08T03:36:30Z)
• Mesh Guided One-shot Face Reenactment using Graph Convolutional Networks [31.083072922977568]
  We introduce a method for one-shot face reenactment, which uses the reconstructed 3D meshes as guidance to learn the optical flow needed for the reenacted face synthesis. We propose a motion net to learn the face motion, which is an asymmetric autoencoder. Our method can generate high-quality results and outperforms state-of-the-art methods in both qualitative and quantitative comparisons.
  arXiv  Detail & Related papers  (2020-08-18T07:41:40Z)
• Head2Head++: Deep Facial Attributes Re-Targeting [6.230979482947681]
  We leverage the 3D geometry of faces and Generative Adversarial Networks (GANs) to design a novel deep learning architecture for the task of facial and head reenactment. We manage to capture the complex non-rigid facial motion from the driving monocular performances and synthesise temporally consistent videos. Our system performs end-to-end reenactment in nearly real-time speed (18 fps)
  arXiv  Detail & Related papers  (2020-06-17T23:38:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.