GM-DF: Generalized Multi-Scenario Deepfake Detection

• URL: http://arxiv.org/abs/2406.20078v1
• Date: Fri, 28 Jun 2024 17:42:08 GMT
• Title: GM-DF: Generalized Multi-Scenario Deepfake Detection
• Authors: Yingxin Lai, Zitong Yu, Jing Yang, Bin Li, Xiangui Kang, Linlin Shen,
• Abstract summary: Existing face forgery detection usually follows the paradigm of training models in a single domain. In this paper, we elaborately investigate the generalization capacity of deepfake detection models when jointly trained on multiple face forgery detection datasets.
• Score: 49.072106087564144
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Existing face forgery detection usually follows the paradigm of training models in a single domain, which leads to limited generalization capacity when unseen scenarios and unknown attacks occur. In this paper, we elaborately investigate the generalization capacity of deepfake detection models when jointly trained on multiple face forgery detection datasets. We first find a rapid degradation of detection accuracy when models are directly trained on combined datasets due to the discrepancy across collection scenarios and generation methods. To address the above issue, a Generalized Multi-Scenario Deepfake Detection framework (GM-DF) is proposed to serve multiple real-world scenarios by a unified model. First, we propose a hybrid expert modeling approach for domain-specific real/forgery feature extraction. Besides, as for the commonality representation, we use CLIP to extract the common features for better aligning visual and textual features across domains. Meanwhile, we introduce a masked image reconstruction mechanism to force models to capture rich forged details. Finally, we supervise the models via a domain-aware meta-learning strategy to further enhance their generalization capacities. Specifically, we design a novel domain alignment loss to strongly align the distributions of the meta-test domains and meta-train domains. Thus, the updated models are able to represent both specific and common real/forgery features across multiple datasets. In consideration of the lack of study of multi-dataset training, we establish a new benchmark leveraging multi-source data to fairly evaluate the models' generalization capacity on unseen scenarios. Both qualitative and quantitative experiments on five datasets conducted on traditional protocols as well as the proposed benchmark demonstrate the effectiveness of our approach.

Related papers

• Learning to Generalize Unseen Domains via Multi-Source Meta Learning for Text Classification [71.08024880298613]
  We study the multi-source Domain Generalization of text classification. We propose a framework to use multiple seen domains to train a model that can achieve high accuracy in an unseen domain.
  arXiv  Detail & Related papers  (2024-09-20T07:46:21Z)
• DG-TTA: Out-of-domain medical image segmentation through Domain Generalization and Test-Time Adaptation [43.842694540544194]
  We propose to combine domain generalization and test-time adaptation to create a highly effective approach for reusing pre-trained models in unseen target domains. We demonstrate that our method, combined with pre-trained whole-body CT models, can effectively segment MR images with high accuracy.
  arXiv  Detail & Related papers  (2023-12-11T10:26:21Z)
• Consistency Regularization for Generalizable Source-free Domain Adaptation [62.654883736925456]
  Source-free domain adaptation (SFDA) aims to adapt a well-trained source model to an unlabelled target domain without accessing the source dataset. Existing SFDA methods ONLY assess their adapted models on the target training set, neglecting the data from unseen but identically distributed testing sets. We propose a consistency regularization framework to develop a more generalizable SFDA method.
  arXiv  Detail & Related papers  (2023-08-03T07:45:53Z)
• Deep Multimodal Fusion for Generalizable Person Re-identification [15.250738959921872]
  DMF is a Deep Multimodal Fusion network for the general scenarios on person re-identification task. Rich semantic knowledge is introduced to assist in feature representation learning during the pre-training stage. A realistic dataset is adopted to fine-tine the pre-trained model for distribution alignment with real-world.
  arXiv  Detail & Related papers  (2022-11-02T07:42:48Z)
• A Novel Mix-normalization Method for Generalizable Multi-source Person Re-identification [49.548815417844786]
  Person re-identification (Re-ID) has achieved great success in the supervised scenario. It is difficult to directly transfer the supervised model to arbitrary unseen domains due to the model overfitting to the seen source domains. We propose MixNorm, which consists of domain-aware mix-normalization (DMN) and domain-ware center regularization (DCR)
  arXiv  Detail & Related papers  (2022-01-24T18:09:38Z)
• Learning to Generalize Unseen Domains via Memory-based Multi-Source Meta-Learning for Person Re-Identification [59.326456778057384]
  We propose the Memory-based Multi-Source Meta-Learning framework to train a generalizable model for unseen domains. We also present a meta batch normalization layer (MetaBN) to diversify meta-test features. Experiments demonstrate that our M$3$L can effectively enhance the generalization ability of the model for unseen domains.
  arXiv  Detail & Related papers  (2020-12-01T11:38:16Z)
• Multi-Domain Adversarial Feature Generalization for Person Re-Identification [52.835955258959785]
  We propose a multi-dataset feature generalization network (MMFA-AAE) It is capable of learning a universal domain-invariant feature representation from multiple labeled datasets and generalizing it to unseen' camera systems. It also surpasses many state-of-the-art supervised methods and unsupervised domain adaptation methods by a large margin.
  arXiv  Detail & Related papers  (2020-11-25T08:03:15Z)
• Generalizable Model-agnostic Semantic Segmentation via Target-specific Normalization [24.14272032117714]
  We propose a novel domain generalization framework for the generalizable semantic segmentation task. We exploit the model-agnostic learning to simulate the domain shift problem. Considering the data-distribution discrepancy between seen source and unseen target domains, we develop the target-specific normalization scheme.
  arXiv  Detail & Related papers  (2020-03-27T09:25:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.