Lips Don't Lie: A Generalisable and Robust Approach to Face Forgery Detection

• URL: http://arxiv.org/abs/2012.07657v2
• Date: Fri, 2 Apr 2021 10:24:56 GMT
• Title: Lips Don't Lie: A Generalisable and Robust Approach to Face Forgery Detection
• Authors: Alexandros Haliassos, Konstantinos Vougioukas, Stavros Petridis, Maja Pantic
• Abstract summary: LipForensics is a detection approach capable of both generalising manipulations and withstanding various distortions. It consists in first pretraining a-temporal network to perform visual speech recognition (lipreading) A temporal network is subsequently finetuned on fixed mouth embeddings of real and forged data in order to detect fake videos based on mouth movements without over-fitting to low-level, manipulation-specific artefacts.
• Score: 118.37239586697139
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Although current deep learning-based face forgery detectors achieve impressive performance in constrained scenarios, they are vulnerable to samples created by unseen manipulation methods. Some recent works show improvements in generalisation but rely on cues that are easily corrupted by common post-processing operations such as compression. In this paper, we propose LipForensics, a detection approach capable of both generalising to novel manipulations and withstanding various distortions. LipForensics targets high-level semantic irregularities in mouth movements, which are common in many generated videos. It consists in first pretraining a spatio-temporal network to perform visual speech recognition (lipreading), thus learning rich internal representations related to natural mouth motion. A temporal network is subsequently finetuned on fixed mouth embeddings of real and forged data in order to detect fake videos based on mouth movements without overfitting to low-level, manipulation-specific artefacts. Extensive experiments show that this simple approach significantly surpasses the state-of-the-art in terms of generalisation to unseen manipulations and robustness to perturbations, as well as shed light on the factors responsible for its performance.

Related papers

• DiffusionFake: Enhancing Generalization in Deepfake Detection via Guided Stable Diffusion [94.46904504076124]
  Deepfake technology has made face swapping highly realistic, raising concerns about the malicious use of fabricated facial content. Existing methods often struggle to generalize to unseen domains due to the diverse nature of facial manipulations. We introduce DiffusionFake, a novel framework that reverses the generative process of face forgeries to enhance the generalization of detection models.
  arXiv  Detail & Related papers  (2024-10-06T06:22:43Z)
• UniForensics: Face Forgery Detection via General Facial Representation [60.5421627990707]
  High-level semantic features are less susceptible to perturbations and not limited to forgery-specific artifacts, thus having stronger generalization. We introduce UniForensics, a novel deepfake detection framework that leverages a transformer-based video network, with a meta-functional face classification for enriched facial representation.
  arXiv  Detail & Related papers  (2024-07-26T20:51:54Z)
• Learning Expressive And Generalizable Motion Features For Face Forgery Detection [52.54404879581527]
  We propose an effective sequence-based forgery detection framework based on an existing video classification method. To make the motion features more expressive for manipulation detection, we propose an alternative motion consistency block. We make a general video classification network achieve promising results on three popular face forgery datasets.
  arXiv  Detail & Related papers  (2024-03-08T09:25:48Z)
• Recap: Detecting Deepfake Video with Unpredictable Tampered Traces via Recovering Faces and Mapping Recovered Faces [35.04806736119123]
  We propose Recap, a novel Deepfake detection model that exposes unspecific facial part inconsistencies by recovering faces. In the recovering stage, the model focuses on randomly masking regions of interest and reconstructing real faces without unpredictable tampered traces. In the mapping stage, the output of the recovery phase serves as supervision to guide the facial mapping process.
  arXiv  Detail & Related papers  (2023-08-19T06:18:11Z)
• Self-supervised Transformer for Deepfake Detection [112.81127845409002]
  Deepfake techniques in real-world scenarios require stronger generalization abilities of face forgery detectors. Inspired by transfer learning, neural networks pre-trained on other large-scale face-related tasks may provide useful features for deepfake detection. In this paper, we propose a self-supervised transformer based audio-visual contrastive learning method.
  arXiv  Detail & Related papers  (2022-03-02T17:44:40Z)
• Leveraging Real Talking Faces via Self-Supervision for Robust Forgery Detection [112.96004727646115]
  We develop a method to detect face-manipulated videos using real talking faces. We show that our method achieves state-of-the-art performance on cross-manipulation generalisation and robustness experiments. Our results suggest that leveraging natural and unlabelled videos is a promising direction for the development of more robust face forgery detectors.
  arXiv  Detail & Related papers  (2022-01-18T17:14:54Z)
• ID-Reveal: Identity-aware DeepFake Video Detection [24.79483180234883]
  ID-Reveal is a new approach that learns temporal facial features, specific of how a person moves while talking. We do not need any training data of fakes, but only train on real videos. We obtain an average improvement of more than 15% in terms of accuracy for facial reenactment on high compressed videos.
  arXiv  Detail & Related papers  (2020-12-04T10:43:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.