AniFaceDiff: High-Fidelity Face Reenactment via Facial Parametric Conditioned Diffusion Models

• URL: http://arxiv.org/abs/2406.13272v1
• Date: Wed, 19 Jun 2024 07:08:48 GMT
• Title: AniFaceDiff: High-Fidelity Face Reenactment via Facial Parametric Conditioned Diffusion Models
• Authors: Ken Chen, Sachith Seneviratne, Wei Wang, Dongting Hu, Sanjay Saha, Md. Tarek Hasan, Sanka Rasnayaka, Tamasha Malepathirana, Mingming Gong, Saman Halgamuge,
• Abstract summary: Face reenactment refers to the process of transferring the pose and facial expressions from a reference (driving) video onto a static facial (source) image. Previous research in this domain has made significant progress by training controllable deep generative models to generate faces. This paper proposes a new method based on Stable Diffusion, called AniFaceDiff, incorporating a new conditioning module for high-fidelity face reenactment.
• Score: 33.39336530229545
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Face reenactment refers to the process of transferring the pose and facial expressions from a reference (driving) video onto a static facial (source) image while maintaining the original identity of the source image. Previous research in this domain has made significant progress by training controllable deep generative models to generate faces based on specific identity, pose and expression conditions. However, the mechanisms used in these methods to control pose and expression often inadvertently introduce identity information from the driving video, while also causing a loss of expression-related details. This paper proposes a new method based on Stable Diffusion, called AniFaceDiff, incorporating a new conditioning module for high-fidelity face reenactment. First, we propose an enhanced 2D facial snapshot conditioning approach by facial shape alignment to prevent the inclusion of identity information from the driving video. Then, we introduce an expression adapter conditioning mechanism to address the potential loss of expression-related information. Our approach effectively preserves pose and expression fidelity from the driving video while retaining the identity and fine details of the source image. Through experiments on the VoxCeleb dataset, we demonstrate that our method achieves state-of-the-art results in face reenactment, showcasing superior image quality, identity preservation, and expression accuracy, especially for cross-identity scenarios. Considering the ethical concerns surrounding potential misuse, we analyze the implications of our method, evaluate current state-of-the-art deepfake detectors, and identify their shortcomings to guide future research.

Related papers

• OSDFace: One-Step Diffusion Model for Face Restoration [72.5045389847792]
  Diffusion models have demonstrated impressive performance in face restoration. We propose OSDFace, a novel one-step diffusion model for face restoration. Results demonstrate that OSDFace surpasses current state-of-the-art (SOTA) methods in both visual quality and quantitative metrics.
  arXiv  Detail & Related papers  (2024-11-26T07:07:48Z)
• DiffusionFace: Towards a Comprehensive Dataset for Diffusion-Based Face Forgery Analysis [71.40724659748787]
  DiffusionFace is the first diffusion-based face forgery dataset. It covers various forgery categories, including unconditional and Text Guide facial image generation, Img2Img, Inpaint, and Diffusion-based facial exchange algorithms. It provides essential metadata and a real-world internet-sourced forgery facial image dataset for evaluation.
  arXiv  Detail & Related papers  (2024-03-27T11:32:44Z)
• DiffusionAct: Controllable Diffusion Autoencoder for One-shot Face Reenactment [34.821255203019554]
  Video-driven neural face reenactment aims to synthesize realistic facial images that successfully preserve the identity and appearance of a source face. Recent advances in Diffusion Probabilistic Models (DPMs) enable the generation of high-quality realistic images. We present Diffusion, a novel method that leverages the photo-realistic image generation of diffusion models to perform neural face reenactment.
  arXiv  Detail & Related papers  (2024-03-25T21:46:53Z)
• When StyleGAN Meets Stable Diffusion: a $\mathscr{W}_+$ Adapter for Personalized Image Generation [60.305112612629465]
  Text-to-image diffusion models have excelled in producing diverse, high-quality, and photo-realistic images. We present a novel use of the extended StyleGAN embedding space $mathcalW_+$ to achieve enhanced identity preservation and disentanglement for diffusion models. Our method adeptly generates personalized text-to-image outputs that are not only compatible with prompt descriptions but also amenable to common StyleGAN editing directions.
  arXiv  Detail & Related papers  (2023-11-29T09:05:14Z)
• HyperReenact: One-Shot Reenactment via Jointly Learning to Refine and Retarget Faces [47.27033282706179]
  We present our method for neural face reenactment, called HyperReenact, that aims to generate realistic talking head images of a source identity. Our method operates under the one-shot setting (i.e., using a single source frame) and allows for cross-subject reenactment, without requiring subject-specific fine-tuning. We compare our method both quantitatively and qualitatively against several state-of-the-art techniques on the standard benchmarks of VoxCeleb1 and VoxCeleb2.
  arXiv  Detail & Related papers  (2023-07-20T11:59:42Z)
• Graph-based Generative Face Anonymisation with Pose Preservation [49.18049578591058]
  AnonyGAN is a GAN-based solution for face anonymisation. It replaces the visual information corresponding to a source identity with a condition identity provided as any single image.
  arXiv  Detail & Related papers  (2021-12-10T12:58:17Z)
• Learning to Aggregate and Personalize 3D Face from In-the-Wild Photo Collection [65.92058628082322]
  Non-parametric face modeling aims to reconstruct 3D face only from images without shape assumptions. This paper presents a novel Learning to Aggregate and Personalize framework for unsupervised robust 3D face modeling.
  arXiv  Detail & Related papers  (2021-06-15T03:10:17Z)
• LI-Net: Large-Pose Identity-Preserving Face Reenactment Network [14.472453602392182]
  We propose a large-pose identity-preserving face reenactment network, LI-Net. Specifically, the Landmark Transformer is adopted to adjust driving landmark images. The Face Rotation Module and the Expression Enhancing Generator decouple the transformed landmark image into pose and expression features, and reenact those attributes separately to generate identity-preserving faces.
  arXiv  Detail & Related papers  (2021-04-07T01:41:21Z)
• VAE/WGAN-Based Image Representation Learning For Pose-Preserving Seamless Identity Replacement In Facial Images [15.855376604558977]
  We present a novel variational generative adversarial network (VGAN) based on Wasserstein loss. We show that our network can be used to perform pose-preserving identity morphing and identity-preserving pose morphing.
  arXiv  Detail & Related papers  (2020-03-02T03:35:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.