DynamicFace: High-Quality and Consistent Video Face Swapping using Composable 3D Facial Priors

• URL: http://arxiv.org/abs/2501.08553v1
• Date: Wed, 15 Jan 2025 03:28:14 GMT
• Title: DynamicFace: High-Quality and Consistent Video Face Swapping using Composable 3D Facial Priors
• Authors: Runqi Wang, Sijie Xu, Tianyao He, Yang Chen, Wei Zhu, Dejia Song, Nemo Chen, Xu Tang, Yao Hu,
• Abstract summary: Face swapping transfers the identity of a source face to a target face while retaining the attributes like expression, pose, hair, and background of the target face. We propose DynamicFace that leverages the power of diffusion model and plug-and-play temporal layers for video face swapping. Our method achieves state-of-the-art results in face swapping, showcasing superior image quality, identity preservation, and expression accuracy.
• Score: 24.721887093958284
• License:
• Abstract: Face swapping transfers the identity of a source face to a target face while retaining the attributes like expression, pose, hair, and background of the target face. Advanced face swapping methods have achieved attractive results. However, these methods often inadvertently transfer identity information from the target face, compromising expression-related details and accurate identity. We propose a novel method DynamicFace that leverages the power of diffusion model and plug-and-play temporal layers for video face swapping. First, we introduce four fine-grained face conditions using 3D facial priors. All conditions are designed to be disentangled from each other for precise and unique control. Then, we adopt Face Former and ReferenceNet for high-level and detailed identity injection. Through experiments on the FF++ dataset, we demonstrate that our method achieves state-of-the-art results in face swapping, showcasing superior image quality, identity preservation, and expression accuracy. Besides, our method could be easily transferred to video domain with temporal attention layer. Our code and results will be available on the project page: https://dynamic-face.github.io/

Related papers

• Towards Consistent and Controllable Image Synthesis for Face Editing [18.646961062736207]
  RigFace is a novel approach to control the lighting, facial expression and head pose of a portrait photo. Our model achieves comparable or even superior performance in both identity preservation and photorealism compared to existing face editing models.
  arXiv  Detail & Related papers  (2025-02-04T16:36:07Z)
• OSDFace: One-Step Diffusion Model for Face Restoration [72.5045389847792]
  Diffusion models have demonstrated impressive performance in face restoration. We propose OSDFace, a novel one-step diffusion model for face restoration. Results demonstrate that OSDFace surpasses current state-of-the-art (SOTA) methods in both visual quality and quantitative metrics.
  arXiv  Detail & Related papers  (2024-11-26T07:07:48Z)
• ReliableSwap: Boosting General Face Swapping Via Reliable Supervision [9.725105108879717]
  This paper proposes to construct reliable supervision, dubbed cycle triplets, which serves as the image-level guidance when the source identity differs from the target one during training. Specifically, we use face reenactment and blending techniques to synthesize the swapped face from real images in advance. Our face swapping framework, named ReliableSwap, can boost the performance of any existing face swapping network with negligible overhead.
  arXiv  Detail & Related papers  (2023-06-08T17:01:14Z)
• Face Transformer: Towards High Fidelity and Accurate Face Swapping [54.737909435708936]
  Face swapping aims to generate swapped images that fuse the identity of source faces and the attributes of target faces. This paper presents Face Transformer, a novel face swapping network that can accurately preserve source identities and target attributes simultaneously.
  arXiv  Detail & Related papers  (2023-04-05T15:51:44Z)
• FlowFace: Semantic Flow-guided Shape-aware Face Swapping [43.166181219154936]
  We propose a semantic flow-guided two-stage framework for shape-aware face swapping, namely FlowFace. Our FlowFace consists of a face reshaping network and a face swapping network. We employ a pre-trained face masked autoencoder to extract facial features from both the source face and the target face.
  arXiv  Detail & Related papers  (2022-12-06T07:23:39Z)
• FaceTuneGAN: Face Autoencoder for Convolutional Expression Transfer Using Neural Generative Adversarial Networks [0.7043489166804575]
  We present FaceTuneGAN, a new 3D face model representation decomposing and encoding separately facial identity and facial expression. We propose a first adaptation of image-to-image translation networks, that have successfully been used in the 2D domain, to 3D face geometry.
  arXiv  Detail & Related papers  (2021-12-01T14:42:03Z)
• HifiFace: 3D Shape and Semantic Prior Guided High Fidelity Face Swapping [116.1022638063613]
  We propose HifiFace, which can preserve the face shape of the source face and generate photo-realistic results. We introduce the Semantic Facial Fusion module to optimize the combination of encoder and decoder features.
  arXiv  Detail & Related papers  (2021-06-18T07:39:09Z)
• Learning to Aggregate and Personalize 3D Face from In-the-Wild Photo Collection [65.92058628082322]
  Non-parametric face modeling aims to reconstruct 3D face only from images without shape assumptions. This paper presents a novel Learning to Aggregate and Personalize framework for unsupervised robust 3D face modeling.
  arXiv  Detail & Related papers  (2021-06-15T03:10:17Z)
• FaceController: Controllable Attribute Editing for Face in the Wild [74.56117807309576]
  We propose a simple feed-forward network to generate high-fidelity manipulated faces. By simply employing some existing and easy-obtainable prior information, our method can control, transfer, and edit diverse attributes of faces in the wild. In our method, we decouple identity, expression, pose, and illumination using 3D priors; separate texture and colors by using region-wise style codes.
  arXiv  Detail & Related papers  (2021-02-23T02:47:28Z)
• Face Super-Resolution Guided by 3D Facial Priors [92.23902886737832]
  We propose a novel face super-resolution method that explicitly incorporates 3D facial priors which grasp the sharp facial structures. Our work is the first to explore 3D morphable knowledge based on the fusion of parametric descriptions of face attributes. The proposed 3D priors achieve superior face super-resolution results over the state-of-the-arts.
  arXiv  Detail & Related papers  (2020-07-18T15:26:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.