$\textit{X}^2$-DFD: A framework for e${X}$plainable and e${X}$tendable Deepfake Detection

• URL: http://arxiv.org/abs/2410.06126v1
• Date: Tue, 8 Oct 2024 15:28:33 GMT
• Title: $\textit{X}^2$-DFD: A framework for e${X}$plainable and e${X}$tendable Deepfake Detection
• Authors: Yize Chen, Zhiyuan Yan, Siwei Lyu, Baoyuan Wu,
• Abstract summary: We propose a novel framework called $X2$-DFD, consisting of three core modules. The first module, Model Feature Assessment (MFA), measures the detection capabilities of forgery features intrinsic to MLLMs, and gives a descending ranking of these features. The second module, Strong Feature Strengthening (SFS), enhances the detection and explanation capabilities by fine-tuning the MLLM on a dataset constructed based on the top-ranked features. The third module, Weak Feature Supplementing (WFS), improves the fine-tuned MLLM's capabilities on lower-ranked features by integrating external dedicated
• Score: 52.14468236527728
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Detecting deepfakes has become an important task. Most existing detection methods provide only real/fake predictions without offering human-comprehensible explanations. Recent studies leveraging MLLMs for deepfake detection have shown improvements in explainability. However, the performance of pre-trained MLLMs (e.g., LLaVA) remains limited due to a lack of understanding of their capabilities for this task and strategies to enhance them. In this work, we empirically assess the strengths and weaknesses of MLLMs specifically in deepfake detection via forgery features analysis. Building on these assessments, we propose a novel framework called ${X}^2$-DFD, consisting of three core modules. The first module, Model Feature Assessment (MFA), measures the detection capabilities of forgery features intrinsic to MLLMs, and gives a descending ranking of these features. The second module, Strong Feature Strengthening (SFS), enhances the detection and explanation capabilities by fine-tuning the MLLM on a dataset constructed based on the top-ranked features. The third module, Weak Feature Supplementing (WFS), improves the fine-tuned MLLM's capabilities on lower-ranked features by integrating external dedicated deepfake detectors. To verify the effectiveness of this framework, we further present a practical implementation, where an automated forgery features generation, evaluation, and ranking procedure is designed for MFA module; an automated generation procedure of the fine-tuning dataset containing real and fake images with explanations based on top-ranked features is developed for SFS model; an external conventional deepfake detector focusing on blending artifact, which corresponds to a low detection capability in the pre-trained MLLM, is integrated for WFS module. Experiments show that our approach enhances both detection and explanation performance.

Related papers

• Efficient Feature Aggregation and Scale-Aware Regression for Monocular 3D Object Detection [40.14197775884804]
  MonoASRH is a novel monocular 3D detection framework composed of Efficient Hybrid Feature Aggregation Module (EH-FAM) and Adaptive Scale-Aware 3D Regression Head (ASRH) EH-FAM employs multi-head attention with a global receptive field to extract semantic features for small-scale objects. ASRH encodes 2D bounding box dimensions and then fuses scale features with the semantic features aggregated by EH-FAM.
  arXiv  Detail & Related papers  (2024-11-05T02:33:25Z)
• MatPlotAgent: Method and Evaluation for LLM-Based Agentic Scientific Data Visualization [86.61052121715689]
  MatPlotAgent is a model-agnostic framework designed to automate scientific data visualization tasks. MatPlotBench is a high-quality benchmark consisting of 100 human-verified test cases.
  arXiv  Detail & Related papers  (2024-02-18T04:28:28Z)
• Fine-Grained Prototypes Distillation for Few-Shot Object Detection [8.795211323408513]
  Few-shot object detection (FSOD) aims at extending a generic detector for novel object detection with only a few training examples. In general, methods based on meta-learning employ an additional support branch to encode novel examples into class prototypes. New methods are required to capture the distinctive local context for more robust novel object detection.
  arXiv  Detail & Related papers  (2024-01-15T12:12:48Z)
• Multistep feature aggregation framework for salient object detection [0.0]
  We introduce a multistep feature aggregation framework for salient object detection. It is composed of three modules, including the Diverse Reception (DR) module, multiscale interaction (MSI) module and Feature Enhancement (FE) module. Experimental results on six benchmark datasets demonstrate that MSFA achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2022-11-12T16:13:16Z)
• The Devil is in the Task: Exploiting Reciprocal Appearance-Localization Features for Monocular 3D Object Detection [62.1185839286255]
  Low-cost monocular 3D object detection plays a fundamental role in autonomous driving. We introduce a Dynamic Feature Reflecting Network, named DFR-Net. We rank 1st among all the monocular 3D object detectors in the KITTI test set.
  arXiv  Detail & Related papers  (2021-12-28T07:31:18Z)
• Progressive Self-Guided Loss for Salient Object Detection [102.35488902433896]
  We present a progressive self-guided loss function to facilitate deep learning-based salient object detection in images. Our framework takes advantage of adaptively aggregated multi-scale features to locate and detect salient objects effectively.
  arXiv  Detail & Related papers  (2021-01-07T07:33:38Z)
• End-to-End Object Detection with Fully Convolutional Network [71.56728221604158]
  We introduce a Prediction-aware One-To-One (POTO) label assignment for classification to enable end-to-end detection. A simple 3D Max Filtering (3DMF) is proposed to utilize the multi-scale features and improve the discriminability of convolutions in the local region. Our end-to-end framework achieves competitive performance against many state-of-the-art detectors with NMS on COCO and CrowdHuman datasets.
  arXiv  Detail & Related papers  (2020-12-07T09:14:55Z)
• iffDetector: Inference-aware Feature Filtering for Object Detection [70.8678270164057]
  We introduce a generic Inference-aware Feature Filtering (IFF) module that can easily be combined with modern detectors. IFF performs closed-loop optimization by leveraging high-level semantics to enhance the convolutional features. IFF can be fused with CNN-based object detectors in a plug-and-play manner with negligible computational cost overhead.
  arXiv  Detail & Related papers  (2020-06-23T02:57:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.