Semantics-Oriented Multitask Learning for DeepFake Detection: A Joint Embedding Approach

• URL: http://arxiv.org/abs/2408.16305v1
• Date: Thu, 29 Aug 2024 07:11:50 GMT
• Title: Semantics-Oriented Multitask Learning for DeepFake Detection: A Joint Embedding Approach
• Authors: Mian Zou, Baosheng Yu, Yibing Zhan, Siwei Lyu, Kede Ma,
• Abstract summary: We propose an automatic dataset expansion technique to support semantics-oriented DeepFake detection tasks. We also resort to joint embedding of face images and their corresponding labels for prediction. Our method improves the generalizability of DeepFake detection and renders some degree of model interpretation by providing human-understandable explanations.
• Score: 77.65459419417533
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In recent years, the multimedia forensics and security community has seen remarkable progress in multitask learning for DeepFake (i.e., face forgery) detection. The prevailing strategy has been to frame DeepFake detection as a binary classification problem augmented by manipulation-oriented auxiliary tasks. This strategy focuses on learning features specific to face manipulations, which exhibit limited generalizability. In this paper, we delve deeper into semantics-oriented multitask learning for DeepFake detection, leveraging the relationships among face semantics via joint embedding. We first propose an automatic dataset expansion technique that broadens current face forgery datasets to support semantics-oriented DeepFake detection tasks at both the global face attribute and local face region levels. Furthermore, we resort to joint embedding of face images and their corresponding labels (depicted by textual descriptions) for prediction. This approach eliminates the need for manually setting task-agnostic and task-specific parameters typically required when predicting labels directly from images. In addition, we employ a bi-level optimization strategy to dynamically balance the fidelity loss weightings of various tasks, making the training process fully automated. Extensive experiments on six DeepFake datasets show that our method improves the generalizability of DeepFake detection and, meanwhile, renders some degree of model interpretation by providing human-understandable explanations.

Related papers

• Leveraging Mixture of Experts for Improved Speech Deepfake Detection [53.69740463004446]
  Speech deepfakes pose a significant threat to personal security and content authenticity. We introduce a novel approach for enhancing speech deepfake detection performance using a Mixture of Experts architecture.
  arXiv  Detail & Related papers  (2024-09-24T13:24:03Z)
• UniForensics: Face Forgery Detection via General Facial Representation [60.5421627990707]
  High-level semantic features are less susceptible to perturbations and not limited to forgery-specific artifacts, thus having stronger generalization. We introduce UniForensics, a novel deepfake detection framework that leverages a transformer-based video network, with a meta-functional face classification for enriched facial representation.
  arXiv  Detail & Related papers  (2024-07-26T20:51:54Z)
• Media Forensics and Deepfake Systematic Survey [0.0]
  Deepfake is a generative deep learning algorithm that creates or changes facial features in a very realistic way. It can be used to make movies look better as well as to spread false information by imitating famous people.
  arXiv  Detail & Related papers  (2024-06-19T07:33:33Z)
• Semantic Contextualization of Face Forgery: A New Definition, Dataset, and Detection Method [77.65459419417533]
  We put face forgery in a semantic context and define that computational methods that alter semantic face attributes are sources of face forgery. We construct a large face forgery image dataset, where each image is associated with a set of labels organized in a hierarchical graph. We propose a semantics-oriented face forgery detection method that captures label relations and prioritizes the primary task.
  arXiv  Detail & Related papers  (2024-05-14T10:24:19Z)
• DeepFidelity: Perceptual Forgery Fidelity Assessment for Deepfake Detection [67.3143177137102]
  Deepfake detection refers to detecting artificially generated or edited faces in images or videos. We propose a novel Deepfake detection framework named DeepFidelity to adaptively distinguish real and fake faces.
  arXiv  Detail & Related papers  (2023-12-07T07:19:45Z)
• Learning to mask: Towards generalized face forgery detection [3.7755650136637304]
  Generalizability to unseen forgery types is crucial for face forgery detectors. Our goal is to reduce the features that are easy to learn in the training phase, so as to reduce the risk of overfitting on specific forgery types. A deep feature mixup strategy is also proposed to synthesize forgeries in the feature domain.
  arXiv  Detail & Related papers  (2022-12-29T13:55:28Z)
• Detecting and Recovering Sequential DeepFake Manipulation [32.34908534582532]
  We propose a novel research problem called Detecting Sequential DeepFake Manipulation (Seq-DeepFake) Unlike the existing deepfake detection task only demanding a binary label prediction, Seq-DeepFake requires correctly predicting a sequential vector of facial manipulation operations. We build a comprehensive benchmark and set up rigorous evaluation protocols and metrics for this new research problem.
  arXiv  Detail & Related papers  (2022-07-05T17:59:33Z)
• Self-supervised Transformer for Deepfake Detection [112.81127845409002]
  Deepfake techniques in real-world scenarios require stronger generalization abilities of face forgery detectors. Inspired by transfer learning, neural networks pre-trained on other large-scale face-related tasks may provide useful features for deepfake detection. In this paper, we propose a self-supervised transformer based audio-visual contrastive learning method.
  arXiv  Detail & Related papers  (2022-03-02T17:44:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.