Lips Don't Lie: A Generalisable and Robust Approach to Face Forgery
Detection
- URL: http://arxiv.org/abs/2012.07657v2
- Date: Fri, 2 Apr 2021 10:24:56 GMT
- Title: Lips Don't Lie: A Generalisable and Robust Approach to Face Forgery
Detection
- Authors: Alexandros Haliassos, Konstantinos Vougioukas, Stavros Petridis, Maja
Pantic
- Abstract summary: LipForensics is a detection approach capable of both generalising manipulations and withstanding various distortions.
It consists in first pretraining a-temporal network to perform visual speech recognition (lipreading)
A temporal network is subsequently finetuned on fixed mouth embeddings of real and forged data in order to detect fake videos based on mouth movements without over-fitting to low-level, manipulation-specific artefacts.
- Score: 118.37239586697139
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Although current deep learning-based face forgery detectors achieve
impressive performance in constrained scenarios, they are vulnerable to samples
created by unseen manipulation methods. Some recent works show improvements in
generalisation but rely on cues that are easily corrupted by common
post-processing operations such as compression. In this paper, we propose
LipForensics, a detection approach capable of both generalising to novel
manipulations and withstanding various distortions. LipForensics targets
high-level semantic irregularities in mouth movements, which are common in many
generated videos. It consists in first pretraining a spatio-temporal network to
perform visual speech recognition (lipreading), thus learning rich internal
representations related to natural mouth motion. A temporal network is
subsequently finetuned on fixed mouth embeddings of real and forged data in
order to detect fake videos based on mouth movements without overfitting to
low-level, manipulation-specific artefacts. Extensive experiments show that
this simple approach significantly surpasses the state-of-the-art in terms of
generalisation to unseen manipulations and robustness to perturbations, as well
as shed light on the factors responsible for its performance.
Related papers
- Learning Expressive And Generalizable Motion Features For Face Forgery
Detection [52.54404879581527]
We propose an effective sequence-based forgery detection framework based on an existing video classification method.
To make the motion features more expressive for manipulation detection, we propose an alternative motion consistency block.
We make a general video classification network achieve promising results on three popular face forgery datasets.
arXiv Detail & Related papers (2024-03-08T09:25:48Z) - Recap: Detecting Deepfake Video with Unpredictable Tampered Traces via
Recovering Faces and Mapping Recovered Faces [35.04806736119123]
We propose Recap, a novel Deepfake detection model that exposes unspecific facial part inconsistencies by recovering faces.
In the recovering stage, the model focuses on randomly masking regions of interest and reconstructing real faces without unpredictable tampered traces.
In the mapping stage, the output of the recovery phase serves as supervision to guide the facial mapping process.
arXiv Detail & Related papers (2023-08-19T06:18:11Z) - Self-supervised Transformer for Deepfake Detection [112.81127845409002]
Deepfake techniques in real-world scenarios require stronger generalization abilities of face forgery detectors.
Inspired by transfer learning, neural networks pre-trained on other large-scale face-related tasks may provide useful features for deepfake detection.
In this paper, we propose a self-supervised transformer based audio-visual contrastive learning method.
arXiv Detail & Related papers (2022-03-02T17:44:40Z) - Leveraging Real Talking Faces via Self-Supervision for Robust Forgery
Detection [112.96004727646115]
We develop a method to detect face-manipulated videos using real talking faces.
We show that our method achieves state-of-the-art performance on cross-manipulation generalisation and robustness experiments.
Our results suggest that leveraging natural and unlabelled videos is a promising direction for the development of more robust face forgery detectors.
arXiv Detail & Related papers (2022-01-18T17:14:54Z) - Watch Those Words: Video Falsification Detection Using Word-Conditioned
Facial Motion [82.06128362686445]
We propose a multi-modal semantic forensic approach to handle both cheapfakes and visually persuasive deepfakes.
We leverage the idea of attribution to learn person-specific biometric patterns that distinguish a given speaker from others.
Unlike existing person-specific approaches, our method is also effective against attacks that focus on lip manipulation.
arXiv Detail & Related papers (2021-12-21T01:57:04Z) - ID-Reveal: Identity-aware DeepFake Video Detection [24.79483180234883]
ID-Reveal is a new approach that learns temporal facial features, specific of how a person moves while talking.
We do not need any training data of fakes, but only train on real videos.
We obtain an average improvement of more than 15% in terms of accuracy for facial reenactment on high compressed videos.
arXiv Detail & Related papers (2020-12-04T10:43:16Z) - Adversarial Attack on Deep Learning-Based Splice Localization [14.669890331986794]
Using a novel algorithm we demonstrate on three non end-to-end deep learning-based splice localization tools that hiding manipulations of images is feasible via adversarial attacks.
We find that the formed adversarial perturbations can be transferable among them regarding the deterioration of their localization performance.
arXiv Detail & Related papers (2020-04-17T20:31:38Z) - Towards Achieving Adversarial Robustness by Enforcing Feature
Consistency Across Bit Planes [51.31334977346847]
We train networks to form coarse impressions based on the information in higher bit planes, and use the lower bit planes only to refine their prediction.
We demonstrate that, by imposing consistency on the representations learned across differently quantized images, the adversarial robustness of networks improves significantly.
arXiv Detail & Related papers (2020-04-01T09:31:10Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.