UniFaceGAN: A Unified Framework for Temporally Consistent Facial Video Editing

• URL: http://arxiv.org/abs/2108.05650v1
• Date: Thu, 12 Aug 2021 10:35:22 GMT
• Title: UniFaceGAN: A Unified Framework for Temporally Consistent Facial Video Editing
• Authors: Meng Cao, Haozhi Huang, Hao Wang, Xuan Wang, Li Shen, Sheng Wang, Linchao Bao, Zhifeng Li, Jiebo Luo
• Abstract summary: We propose a unified temporally consistent facial video editing framework termed UniFaceGAN. Our framework is designed to handle face swapping and face reenactment simultaneously. Compared with the state-of-the-art facial image editing methods, our framework generates video portraits that are more photo-realistic and temporally smooth.
• Score: 78.26925404508994
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent research has witnessed advances in facial image editing tasks including face swapping and face reenactment. However, these methods are confined to dealing with one specific task at a time. In addition, for video facial editing, previous methods either simply apply transformations frame by frame or utilize multiple frames in a concatenated or iterative fashion, which leads to noticeable visual flickers. In this paper, we propose a unified temporally consistent facial video editing framework termed UniFaceGAN. Based on a 3D reconstruction model and a simple yet efficient dynamic training sample selection mechanism, our framework is designed to handle face swapping and face reenactment simultaneously. To enforce the temporal consistency, a novel 3D temporal loss constraint is introduced based on the barycentric coordinate interpolation. Besides, we propose a region-aware conditional normalization layer to replace the traditional AdaIN or SPADE to synthesize more context-harmonious results. Compared with the state-of-the-art facial image editing methods, our framework generates video portraits that are more photo-realistic and temporally smooth.

Related papers

• Kalman-Inspired Feature Propagation for Video Face Super-Resolution [78.84881180336744]
  We introduce a novel framework to maintain a stable face prior to time. The Kalman filtering principles offer our method a recurrent ability to use the information from previously restored frames to guide and regulate the restoration process of the current frame. Experiments demonstrate the effectiveness of our method in capturing facial details consistently across video frames.
  arXiv  Detail & Related papers  (2024-08-09T17:57:12Z)
• FED-NeRF: Achieve High 3D Consistency and Temporal Coherence for Face Video Editing on Dynamic NeRF [77.94545888842883]
  This paper proposes a novel face video editing architecture built upon the dynamic face GAN-NeRF structure. By editing the latent code, multi-view consistent editing on the face can be ensured, as validated by multiview stereo reconstruction. We propose a stabilizer that maintains temporal coherence by preserving smooth changes of face expressions in consecutive frames.
  arXiv  Detail & Related papers  (2024-01-05T03:23:38Z)
• Diffusion Video Autoencoders: Toward Temporally Consistent Face Video Editing via Disentangled Video Encoding [35.18070525015657]
  We propose a novel face video editing framework based on diffusion autoencoders. Our model is based on diffusion models and can satisfy both reconstruction and edit capabilities at the same time.
  arXiv  Detail & Related papers  (2022-12-06T07:41:51Z)
• Video2StyleGAN: Encoding Video in Latent Space for Manipulation [63.03250800510085]
  We propose a novel network to encode face videos into the latent space of StyleGAN for semantic face video manipulation. Our approach can significantly outperform existing single image methods, while achieving real-time (66 fps) speed.
  arXiv  Detail & Related papers  (2022-06-27T06:48:15Z)
• F3A-GAN: Facial Flow for Face Animation with Generative Adversarial Networks [24.64246570503213]
  We propose a novel representation based on a 3D geometric flow, termed facial flow, to represent the natural motion of the human face at any pose. In order to utilize the facial flow for face editing, we build a framework generating continuous images with conditional facial flows.
  arXiv  Detail & Related papers  (2022-05-12T16:40:27Z)
• Image-to-Video Generation via 3D Facial Dynamics [78.01476554323179]
  We present a versatile model, FaceAnime, for various video generation tasks from still images. Our model is versatile for various AR/VR and entertainment applications, such as face video and face video prediction.
  arXiv  Detail & Related papers  (2021-05-31T02:30:11Z)
• Task-agnostic Temporally Consistent Facial Video Editing [84.62351915301795]
  We propose a task-agnostic, temporally consistent facial video editing framework. Based on a 3D reconstruction model, our framework is designed to handle several editing tasks in a more unified and disentangled manner. Compared with the state-of-the-art facial image editing methods, our framework generates video portraits that are more photo-realistic and temporally smooth.
  arXiv  Detail & Related papers  (2020-07-03T02:49:20Z)
• Head2Head++: Deep Facial Attributes Re-Targeting [6.230979482947681]
  We leverage the 3D geometry of faces and Generative Adversarial Networks (GANs) to design a novel deep learning architecture for the task of facial and head reenactment. We manage to capture the complex non-rigid facial motion from the driving monocular performances and synthesise temporally consistent videos. Our system performs end-to-end reenactment in nearly real-time speed (18 fps)
  arXiv  Detail & Related papers  (2020-06-17T23:38:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.