TAR: Generalized Forensic Framework to Detect Deepfakes using Weakly
Supervised Learning
- URL: http://arxiv.org/abs/2105.06117v1
- Date: Thu, 13 May 2021 07:31:08 GMT
- Title: TAR: Generalized Forensic Framework to Detect Deepfakes using Weakly
Supervised Learning
- Authors: Sangyup Lee, Shahroz Tariq, Junyaup Kim, and Simon S. Woo
- Abstract summary: Deepfakes have become a critical social problem, and detecting them is of utmost importance.
In this work, we introduce a practical digital forensic tool to detect different types of deepfakes simultaneously.
We develop an autoencoder-based detection model with Residual blocks and sequentially perform transfer learning to detect different types of deepfakes simultaneously.
- Score: 17.40885531847159
- License: http://creativecommons.org/licenses/by-nc-nd/4.0/
- Abstract: Deepfakes have become a critical social problem, and detecting them is of
utmost importance. Also, deepfake generation methods are advancing, and it is
becoming harder to detect. While many deepfake detection models can detect
different types of deepfakes separately, they perform poorly on generalizing
the detection performance over multiple types of deepfake. This motivates us to
develop a generalized model to detect different types of deepfakes. Therefore,
in this work, we introduce a practical digital forensic tool to detect
different types of deepfakes simultaneously and propose Transfer learning-based
Autoencoder with Residuals (TAR). The ultimate goal of our work is to develop a
unified model to detect various types of deepfake videos with high accuracy,
with only a small number of training samples that can work well in real-world
settings. We develop an autoencoder-based detection model with Residual blocks
and sequentially perform transfer learning to detect different types of
deepfakes simultaneously. Our approach achieves a much higher generalized
detection performance than the state-of-the-art methods on the FaceForensics++
dataset. In addition, we evaluate our model on 200 real-world
Deepfake-in-the-Wild (DW) videos of 50 celebrities available on the Internet
and achieve 89.49% zero-shot accuracy, which is significantly higher than the
best baseline model (gaining 10.77%), demonstrating and validating the
practicability of our approach.
Related papers
- DF40: Toward Next-Generation Deepfake Detection [62.073997142001424]
Existing works identify top-notch detection algorithms and models by adhering to the common practice: training detectors on one specific dataset (e.g., FF++) and testing them on other prevalent deepfake datasets.
But can these stand-out "winners" be truly applied to tackle the myriad of realistic and diverse deepfakes lurking in the real world?
We construct a highly diverse and large-scale deepfake dataset called DF40, which comprises 40 distinct deepfake techniques.
We then conduct comprehensive evaluations using 4 standard evaluation protocols and 7 representative detectors, resulting in over 2,000 evaluations.
arXiv Detail & Related papers (2024-06-19T12:35:02Z) - AntifakePrompt: Prompt-Tuned Vision-Language Models are Fake Image
Detectors [27.07771989900852]
Deep generative models can create remarkably fake images while raising concerns about misinformation and copyright infringement.
Deepfake detection technique is developed to distinguish between real and fake images.
We propose a novel approach using Vision-Language Models (VLMs) and prompt tuning techniques to improve the deepfake detection accuracy over unseen data.
arXiv Detail & Related papers (2023-10-26T14:23:45Z) - Improving Cross-dataset Deepfake Detection with Deep Information
Decomposition [57.284370468207214]
Deepfake technology poses a significant threat to security and social trust.
Existing detection methods suffer from sharp performance degradation when faced with cross-dataset scenarios.
We propose a deep information decomposition (DID) framework in this paper.
arXiv Detail & Related papers (2023-09-30T12:30:25Z) - How Generalizable are Deepfake Detectors? An Empirical Study [5.380277044998179]
We present the first empirical study on the generalizability of deepfake detectors.
Our study utilizes six deepfake datasets, five deepfake detection methods, and two model augmentation approaches.
arXiv Detail & Related papers (2023-08-08T10:30:34Z) - Deep Convolutional Pooling Transformer for Deepfake Detection [54.10864860009834]
We propose a deep convolutional Transformer to incorporate decisive image features both locally and globally.
Specifically, we apply convolutional pooling and re-attention to enrich the extracted features and enhance efficacy.
The proposed solution consistently outperforms several state-of-the-art baselines on both within- and cross-dataset experiments.
arXiv Detail & Related papers (2022-09-12T15:05:41Z) - A Continual Deepfake Detection Benchmark: Dataset, Methods, and
Essentials [97.69553832500547]
This paper suggests a continual deepfake detection benchmark (CDDB) over a new collection of deepfakes from both known and unknown generative models.
We exploit multiple approaches to adapt multiclass incremental learning methods, commonly used in the continual visual recognition, to the continual deepfake detection problem.
arXiv Detail & Related papers (2022-05-11T13:07:19Z) - Metamorphic Testing-based Adversarial Attack to Fool Deepfake Detectors [2.0649235321315285]
There is a dire need for deepfake detection technology to help spot deepfake media.
Current deepfake detection models are able to achieve outstanding accuracy (>90%)
This study identifies makeup application as an adversarial attack that could fool deepfake detectors.
arXiv Detail & Related papers (2022-04-19T02:24:30Z) - Self-supervised Transformer for Deepfake Detection [112.81127845409002]
Deepfake techniques in real-world scenarios require stronger generalization abilities of face forgery detectors.
Inspired by transfer learning, neural networks pre-trained on other large-scale face-related tasks may provide useful features for deepfake detection.
In this paper, we propose a self-supervised transformer based audio-visual contrastive learning method.
arXiv Detail & Related papers (2022-03-02T17:44:40Z) - Model Attribution of Face-swap Deepfake Videos [39.771800841412414]
We first introduce a new dataset with DeepFakes from Different Models (DFDM) based on several Autoencoder models.
Specifically, five generation models with variations in encoder, decoder, intermediate layer, input resolution, and compression ratio have been used to generate a total of 6,450 Deepfake videos.
We take Deepfakes model attribution as a multiclass classification task and propose a spatial and temporal attention based method to explore the differences among Deepfakes.
arXiv Detail & Related papers (2022-02-25T20:05:18Z) - One Detector to Rule Them All: Towards a General Deepfake Attack
Detection Framework [19.762839181838388]
We introduce a Convolutional LSTM-based Residual Network (CLRNet) to better cope with unknown and unseen deepfakes.
Our CLRNet model demonstrated that it generalizes well against high-quality DFW videos by achieving 93.86% detection accuracy.
arXiv Detail & Related papers (2021-05-01T08:02:59Z) - M2TR: Multi-modal Multi-scale Transformers for Deepfake Detection [74.19291916812921]
forged images generated by Deepfake techniques pose a serious threat to the trustworthiness of digital information.
In this paper, we aim to capture the subtle manipulation artifacts at different scales for Deepfake detection.
We introduce a high-quality Deepfake dataset, SR-DF, which consists of 4,000 DeepFake videos generated by state-of-the-art face swapping and facial reenactment methods.
arXiv Detail & Related papers (2021-04-20T05:43:44Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.