DeepFake-Adapter: Dual-Level Adapter for DeepFake Detection

• URL: http://arxiv.org/abs/2306.00863v1
• Date: Thu, 1 Jun 2023 16:23:22 GMT
• Title: DeepFake-Adapter: Dual-Level Adapter for DeepFake Detection
• Authors: Rui Shao, Tianxing Wu, Liqiang Nie, Ziwei Liu
• Abstract summary: Existing deepfake detection methods fail to generalize well to unseen or degraded samples. High-level semantics are indispensable recipes for generalizable forgery detection. DeepFake-Adapter is first parameter-efficient tuning approach for deepfake detection.
• Score: 73.66077273888018
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Existing deepfake detection methods fail to generalize well to unseen or degraded samples, which can be attributed to the over-fitting of low-level forgery patterns. Here we argue that high-level semantics are also indispensable recipes for generalizable forgery detection. Recently, large pre-trained Vision Transformers (ViTs) have shown promising generalization capability. In this paper, we propose the first parameter-efficient tuning approach for deepfake detection, namely DeepFake-Adapter, to effectively and efficiently adapt the generalizable high-level semantics from large pre-trained ViTs to aid deepfake detection. Given large pre-trained models but limited deepfake data, DeepFake-Adapter introduces lightweight yet dedicated dual-level adapter modules to a ViT while keeping the model backbone frozen. Specifically, to guide the adaptation process to be aware of both global and local forgery cues of deepfake data, 1) we not only insert Globally-aware Bottleneck Adapters in parallel to MLP layers of ViT, 2) but also actively cross-attend Locally-aware Spatial Adapters with features from ViT. Unlike existing deepfake detection methods merely focusing on low-level forgery patterns, the forgery detection process of our model can be regularized by generalizable high-level semantics from a pre-trained ViT and adapted by global and local low-level forgeries of deepfake data. Extensive experiments on several standard deepfake detection benchmarks validate the effectiveness of our approach. Notably, DeepFake-Adapter demonstrates a convincing advantage under cross-dataset and cross-manipulation settings. The source code is released at https://github.com/rshaojimmy/DeepFake-Adapter

Related papers

• FakeFormer: Efficient Vulnerability-Driven Transformers for Generalisable Deepfake Detection [12.594436202557446]
  This paper investigates why Vision Transformers (ViTs) exhibit a suboptimal performance when dealing with the detection of facial forgeries. We propose a deepfake detection framework called FakeFormer, which extends ViTs to enforce the extraction of subtle inconsistency-prone information. Experiments are conducted on diverse well-known datasets, including FF++, Celeb-DF, WildDeepfake, DFD, DFDCP, and DFDC.
  arXiv  Detail & Related papers  (2024-10-29T11:36:49Z)
• Harnessing Wavelet Transformations for Generalizable Deepfake Forgery Detection [0.0]
  Wavelet-CLIP is a deepfake detection framework that integrates wavelet transforms with features derived from the ViT-L/14 architecture, pre-trained in the CLIP fashion. Our method showcases outstanding performance, achieving an average AUC of 0.749 for cross-data generalization and 0.893 for robustness against unseen deepfakes.
  arXiv  Detail & Related papers  (2024-09-26T21:16:51Z)
• Standing on the Shoulders of Giants: Reprogramming Visual-Language Model for General Deepfake Detection [16.21235742118949]
  We propose a novel approach that repurposes a well-trained Vision-Language Models (VLMs) for general deepfake detection. Motivated by the model reprogramming paradigm that manipulates the model prediction via data perturbations, our method can reprogram a pretrained VLM model. Our superior performances are at less cost of trainable parameters, making it a promising approach for real-world applications.
  arXiv  Detail & Related papers  (2024-09-04T12:46:30Z)
• Semantics-Oriented Multitask Learning for DeepFake Detection: A Joint Embedding Approach [77.65459419417533]
  We propose an automatic dataset expansion technique to support semantics-oriented DeepFake detection tasks. We also resort to joint embedding of face images and their corresponding labels for prediction. Our method improves the generalizability of DeepFake detection and renders some degree of model interpretation by providing human-understandable explanations.
  arXiv  Detail & Related papers  (2024-08-29T07:11:50Z)
• Open-Set Deepfake Detection: A Parameter-Efficient Adaptation Method with Forgery Style Mixture [58.60915132222421]
  We introduce an approach that is both general and parameter-efficient for face forgery detection. We design a forgery-style mixture formulation that augments the diversity of forgery source domains. We show that the designed model achieves state-of-the-art generalizability with significantly reduced trainable parameters.
  arXiv  Detail & Related papers  (2024-08-23T01:53:36Z)
• MoE-FFD: Mixture of Experts for Generalized and Parameter-Efficient Face Forgery Detection [54.545054873239295]
  Deepfakes have recently raised significant trust issues and security concerns among the public. ViT-based methods take advantage of the expressivity of transformers, achieving superior detection performance. This work introduces Mixture-of-Experts modules for Face Forgery Detection (MoE-FFD), a generalized yet parameter-efficient ViT-based approach.
  arXiv  Detail & Related papers  (2024-04-12T13:02:08Z)
• Generalized Face Forgery Detection via Adaptive Learning for Pre-trained Vision Transformer [54.32283739486781]
  We present a textbfForgery-aware textbfAdaptive textbfVision textbfTransformer (FA-ViT) under the adaptive learning paradigm. FA-ViT achieves 93.83% and 78.32% AUC scores on Celeb-DF and DFDC datasets in the cross-dataset evaluation.
  arXiv  Detail & Related papers  (2023-09-20T06:51:11Z)
• Delving into Sequential Patches for Deepfake Detection [64.19468088546743]
  Recent advances in face forgery techniques produce nearly untraceable deepfake videos, which could be leveraged with malicious intentions. Previous studies has identified the importance of local low-level cues and temporal information in pursuit to generalize well across deepfake methods. We propose the Local- & Temporal-aware Transformer-based Deepfake Detection framework, which adopts a local-to-global learning protocol.
  arXiv  Detail & Related papers  (2022-07-06T16:46:30Z)
• TAR: Generalized Forensic Framework to Detect Deepfakes using Weakly Supervised Learning [17.40885531847159]
  Deepfakes have become a critical social problem, and detecting them is of utmost importance. In this work, we introduce a practical digital forensic tool to detect different types of deepfakes simultaneously. We develop an autoencoder-based detection model with Residual blocks and sequentially perform transfer learning to detect different types of deepfakes simultaneously.
  arXiv  Detail & Related papers  (2021-05-13T07:31:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.