WWW: Where, Which and Whatever Enhancing Interpretability in Multimodal Deepfake Detection

• URL: http://arxiv.org/abs/2408.02954v1
• Date: Tue, 6 Aug 2024 04:44:10 GMT
• Title: WWW: Where, Which and Whatever Enhancing Interpretability in Multimodal Deepfake Detection
• Abstract summary: Current benchmarks for multimodal deepfake detection manipulate entire frames using various generation techniques. These benchmarks struggle to detect dynamic deepfake attacks with challenging frame-by-frame alterations presented in real-world scenarios. We introduce FakeMix, a novel clip-level evaluation benchmark aimed at identifying manipulated segments within both video and audio.
• Score: 10.770226158965686
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: All current benchmarks for multimodal deepfake detection manipulate entire frames using various generation techniques, resulting in oversaturated detection accuracies exceeding 94% at the video-level classification. However, these benchmarks struggle to detect dynamic deepfake attacks with challenging frame-by-frame alterations presented in real-world scenarios. To address this limitation, we introduce FakeMix, a novel clip-level evaluation benchmark aimed at identifying manipulated segments within both video and audio, providing insight into the origins of deepfakes. Furthermore, we propose novel evaluation metrics, Temporal Accuracy (TA) and Frame-wise Discrimination Metric (FDM), to assess the robustness of deepfake detection models. Evaluating state-of-the-art models against diverse deepfake benchmarks, particularly FakeMix, demonstrates the effectiveness of our approach comprehensively. Specifically, while achieving an Average Precision (AP) of 94.2% at the video-level, the evaluation of the existing models at the clip-level using the proposed metrics, TA and FDM, yielded sharp declines in accuracy to 53.1%, and 52.1%, respectively.

Related papers

• FakeI2V-Bench: Benchmarking the Applicability of Image-level Deepfake Detectors for Deepfake Video Detection [15.79371319503651]
  FakeI2V-Bench is a benchmark for evaluating state-of-the-art video-level deepfake detectors.<n> IV-Bridge is a framework that enhances the applicability of image-level deepfake detectors to videos.
  arXiv  Detail & Related papers  (2026-08-04T04:13:31Z)
• LoCC: Detection and Localization of Lip-Syncing Deepfakes via Counterfactual Frame Consistency [28.817029705247155]
  Lip-syncing deepfakes are among the most challenging forms of manipulated media because their artifacts are localized almost exclusively to the mouth region.<n>We propose LoCC, a novel detection framework that performs fine-grained detection and localization of lip-syncing deepfakes at both segment and frame levels.
  arXiv  Detail & Related papers  (2026-06-22T02:19:39Z)
• MSAVBench: Towards Comprehensive and Reliable Evaluation of Multi-Shot Audio-Video Generation [88.7702943548674]
  We introduce MSAVBench, the first comprehensive benchmark and adaptive hybrid evaluation framework for multi-shot audio-video generation.<n>Our benchmark spans four key dimensions, video, audio, shot, and reference, covering diverse task settings, varying shot counts of up to 15, and challenging non-realistic scenarios.<n> MSAVBench achieves high alignment with human judgments, reaching a Spearman rank correlation of 91.5%.
  arXiv  Detail & Related papers  (2026-05-19T17:59:33Z)
• Beyond Raw Detection Scores: Markov-Informed Calibration for Boosting Machine-Generated Text Detection [105.14032334647932]
  Machine-generated texts (MGTs) pose risks such as disinformation and phishing, highlighting the need for reliable detection.<n> Metric-based methods, which extract statistically distinguishable features of MGTs, are often more practical than complex model-based methods that are prone to overfitting.<n>We propose a Markov-informed score calibration strategy that models two relationships of context detection scores that may aid calibration.
  arXiv  Detail & Related papers  (2026-02-08T16:06:12Z)
• FakeParts: a New Family of AI-Generated DeepFakes [20.563685866398384]
  We introduce FakeParts, a new class of deepfakes characterized by subtle, localized manipulations to specific spatial regions or temporal segments of otherwise authentic videos.<n>We present FakePartsBench, the first large-scale benchmark dataset specifically designed to capture the full spectrum of partial deepfakes.<n>Our user studies demonstrate that FakeParts reduces human detection accuracy by over 30% compared to traditional deepfakes, with similar performance degradation observed in state-of-the-art detection models.
  arXiv  Detail & Related papers  (2025-08-28T17:55:14Z)
• UniForensics: Face Forgery Detection via General Facial Representation [60.5421627990707]
  High-level semantic features are less susceptible to perturbations and not limited to forgery-specific artifacts, thus having stronger generalization. We introduce UniForensics, a novel deepfake detection framework that leverages a transformer-based video network, with a meta-functional face classification for enriched facial representation.
  arXiv  Detail & Related papers  (2024-07-26T20:51:54Z)
• CLIPping the Deception: Adapting Vision-Language Models for Universal Deepfake Detection [3.849401956130233]
  We explore the effectiveness of pre-trained vision-language models (VLMs) when paired with recent adaptation methods for universal deepfake detection. We employ only a single dataset (ProGAN) in order to adapt CLIP for deepfake detection. The simple and lightweight Prompt Tuning based adaptation strategy outperforms the previous SOTA approach by 5.01% mAP and 6.61% accuracy.
  arXiv  Detail & Related papers  (2024-02-20T11:26:42Z)
• Facial Forgery-based Deepfake Detection using Fine-Grained Features [7.378937711027777]
  Facial forgery by deepfakes has caused major security risks and raised severe societal concerns. We formulate deepfake detection as a fine-grained classification problem and propose a new fine-grained solution to it. Our method is based on learning subtle and generalizable features by effectively suppressing background noise and learning discriminative features at various scales for deepfake detection.
  arXiv  Detail & Related papers  (2023-10-10T21:30:05Z)
• CrossDF: Improving Cross-Domain Deepfake Detection with Deep Information Decomposition [53.860796916196634]
  We propose a Deep Information Decomposition (DID) framework to enhance the performance of Cross-dataset Deepfake Detection (CrossDF) Unlike most existing deepfake detection methods, our framework prioritizes high-level semantic features over specific visual artifacts. It adaptively decomposes facial features into deepfake-related and irrelevant information, only using the intrinsic deepfake-related information for real/fake discrimination.
  arXiv  Detail & Related papers  (2023-09-30T12:30:25Z)
• Towards Generalizable Deepfake Detection by Primary Region Regularization [52.41801719896089]
  This paper enhances the generalization capability from a novel regularization perspective. Our method consists of two stages, namely the static localization for primary region maps, and the dynamic exploitation of primary region masks. We conduct extensive experiments over three widely used deepfake datasets - DFDC, DF-1.0, and Celeb-DF with five backbones.
  arXiv  Detail & Related papers  (2023-07-24T05:43:34Z)
• DeepfakeBench: A Comprehensive Benchmark of Deepfake Detection [55.70982767084996]
  A critical yet frequently overlooked challenge in the field of deepfake detection is the lack of a standardized, unified, comprehensive benchmark. We present the first comprehensive benchmark for deepfake detection, called DeepfakeBench, which offers three key contributions. DeepfakeBench contains 15 state-of-the-art detection methods, 9CL datasets, a series of deepfake detection evaluation protocols and analysis tools, as well as comprehensive evaluations.
  arXiv  Detail & Related papers  (2023-07-04T01:34:41Z)
• Assessment Framework for Deepfake Detection in Real-world Situations [13.334500258498798]
  Deep learning-based deepfake detection methods have exhibited remarkable performance. The impact of various image and video processing operations and typical workflow distortions on detection accuracy has not been systematically measured. A more reliable assessment framework is proposed to evaluate the performance of learning-based deepfake detectors in more realistic settings.
  arXiv  Detail & Related papers  (2023-04-12T19:09:22Z)
• A Continual Deepfake Detection Benchmark: Dataset, Methods, and Essentials [97.69553832500547]
  This paper suggests a continual deepfake detection benchmark (CDDB) over a new collection of deepfakes from both known and unknown generative models. We exploit multiple approaches to adapt multiclass incremental learning methods, commonly used in the continual visual recognition, to the continual deepfake detection problem.
  arXiv  Detail & Related papers  (2022-05-11T13:07:19Z)
• On Improving Cross-dataset Generalization of Deepfake Detectors [1.0152838128195467]
  Facial manipulation by deep fake has caused major security risks and raised severe societal concerns. We formulate deep fake detection as a hybrid combination of supervised and reinforcement learning (RL) to improve its cross-dataset generalization performance. We demonstrate the superiority of our method over existing published research in cross-dataset generalization of deep fake detectors, thus obtaining state-of-the-art performance.
  arXiv  Detail & Related papers  (2022-04-08T20:34:53Z)
• Voice-Face Homogeneity Tells Deepfake [56.334968246631725]
  Existing detection approaches contribute to exploring the specific artifacts in deepfake videos. We propose to perform the deepfake detection from an unexplored voice-face matching view. Our model obtains significantly improved performance as compared to other state-of-the-art competitors.
  arXiv  Detail & Related papers  (2022-03-04T09:08:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.