Facial Reenactment Through a Personalized Generator

• URL: http://arxiv.org/abs/2307.06307v1
• Date: Wed, 12 Jul 2023 17:09:18 GMT
• Title: Facial Reenactment Through a Personalized Generator
• Authors: Ariel Elazary, Yotam Nitzan, Daniel Cohen-Or
• Abstract summary: We propose a novel method for facial reenactment using a personalized generator. We locate the desired frames in the latent space of the personalized generator using carefully designed latent optimization. We show that since our reenactment takes place in a semantic latent space, it can be semantically edited and stylized in post-processing.
• Score: 47.02774886256621
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In recent years, the role of image generative models in facial reenactment has been steadily increasing. Such models are usually subject-agnostic and trained on domain-wide datasets. The appearance of the reenacted individual is learned from a single image, and hence, the entire breadth of the individual's appearance is not entirely captured, leading these methods to resort to unfaithful hallucination. Thanks to recent advancements, it is now possible to train a personalized generative model tailored specifically to a given individual. In this paper, we propose a novel method for facial reenactment using a personalized generator. We train the generator using frames from a short, yet varied, self-scan video captured using a simple commodity camera. Images synthesized by the personalized generator are guaranteed to preserve identity. The premise of our work is that the task of reenactment is thus reduced to accurately mimicking head poses and expressions. To this end, we locate the desired frames in the latent space of the personalized generator using carefully designed latent optimization. Through extensive evaluation, we demonstrate state-of-the-art performance for facial reenactment. Furthermore, we show that since our reenactment takes place in a semantic latent space, it can be semantically edited and stylized in post-processing.

Related papers

• When StyleGAN Meets Stable Diffusion: a $\mathscr{W}_+$ Adapter for Personalized Image Generation [60.305112612629465]
  Text-to-image diffusion models have excelled in producing diverse, high-quality, and photo-realistic images. We present a novel use of the extended StyleGAN embedding space $mathcalW_+$ to achieve enhanced identity preservation and disentanglement for diffusion models. Our method adeptly generates personalized text-to-image outputs that are not only compatible with prompt descriptions but also amenable to common StyleGAN editing directions.
  arXiv  Detail & Related papers  (2023-11-29T09:05:14Z)
• HyperReenact: One-Shot Reenactment via Jointly Learning to Refine and Retarget Faces [47.27033282706179]
  We present our method for neural face reenactment, called HyperReenact, that aims to generate realistic talking head images of a source identity. Our method operates under the one-shot setting (i.e., using a single source frame) and allows for cross-subject reenactment, without requiring subject-specific fine-tuning. We compare our method both quantitatively and qualitatively against several state-of-the-art techniques on the standard benchmarks of VoxCeleb1 and VoxCeleb2.
  arXiv  Detail & Related papers  (2023-07-20T11:59:42Z)
• Identity-Preserving Talking Face Generation with Landmark and Appearance Priors [106.79923577700345]
  Existing person-generic methods have difficulty in generating realistic and lip-synced videos. We propose a two-stage framework consisting of audio-to-landmark generation and landmark-to-video rendering procedures. Our method can produce more realistic, lip-synced, and identity-preserving videos than existing person-generic talking face generation methods.
  arXiv  Detail & Related papers  (2023-05-15T01:31:32Z)
• Attribute-preserving Face Dataset Anonymization via Latent Code Optimization [64.4569739006591]
  We present a task-agnostic anonymization procedure that directly optimize the images' latent representation in the latent space of a pre-trained GAN. We demonstrate through a series of experiments that our method is capable of anonymizing the identity of the images whilst -- crucially -- better-preserving the facial attributes.
  arXiv  Detail & Related papers  (2023-03-20T17:34:05Z)
• StyleMask: Disentangling the Style Space of StyleGAN2 for Neural Face Reenactment [47.27033282706179]
  We propose a framework that learns to disentangle the identity characteristics of the face from its pose. We show that the proposed method produces higher quality results even on extreme pose variations.
  arXiv  Detail & Related papers  (2022-09-27T13:22:35Z)
• MyStyle: A Personalized Generative Prior [38.3436972491162]
  We introduce MyStyle, a personalized deep generative prior trained with a few shots of an individual. MyStyle allows to reconstruct, enhance and edit images of a specific person.
  arXiv  Detail & Related papers  (2022-03-31T17:59:19Z)
• One-Shot Identity-Preserving Portrait Reenactment [16.889479797252783]
  We present a deep learning-based framework for portrait reenactment from a single picture of a target (one-shot) and a video of a driving subject. We aim to address identity preservation in cross-subject portrait reenactment from a single picture.
  arXiv  Detail & Related papers  (2020-04-26T18:30:33Z)
• ActGAN: Flexible and Efficient One-shot Face Reenactment [1.8431600219151503]
  ActGAN is a novel end-to-end generative adversarial network (GAN) for one-shot face reenactment. We introduce a "many-to-many" approach, which allows arbitrary persons both for source and target without additional retraining. We also introduce a solution to preserve a person's identity between synthesized and target person by adopting the state-of-the-art approach in deep face recognition domain.
  arXiv  Detail & Related papers  (2020-03-30T22:03:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.