Evaluating Deepfake Detectors in the Wild

• URL: http://arxiv.org/abs/2507.21905v2
• Date: Mon, 04 Aug 2025 13:19:11 GMT
• Title: Evaluating Deepfake Detectors in the Wild
• Authors: Viacheslav Pirogov, Maksim Artemev,
• Abstract summary: We evaluate modern deepfake detectors, introducing a novel testing procedure designed to mimic real-world scenarios for deepfake detection.<n>Our analysis shows that detecting deepfakes still remains a challenging task.<n>Basic image manipulations, such as JPEG compression or image enhancement, can significantly reduce model performance.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Deepfakes powered by advanced machine learning models present a significant and evolving threat to identity verification and the authenticity of digital media. Although numerous detectors have been developed to address this problem, their effectiveness has yet to be tested when applied to real-world data. In this work we evaluate modern deepfake detectors, introducing a novel testing procedure designed to mimic real-world scenarios for deepfake detection. Using state-of-the-art deepfake generation methods, we create a comprehensive dataset containing more than 500,000 high-quality deepfake images. Our analysis shows that detecting deepfakes still remains a challenging task. The evaluation shows that in fewer than half of the deepfake detectors tested achieved an AUC score greater than 60%, with the lowest being 50%. We demonstrate that basic image manipulations, such as JPEG compression or image enhancement, can significantly reduce model performance. All code and data are publicly available at https://github.com/SumSubstance/Deepfake-Detectors-in-the-Wild.

Related papers

• Pay Less Attention to Deceptive Artifacts: Robust Detection of Compressed Deepfakes on Online Social Networks [81.21729774122554]
  Existing deepfake detection methods overlook the block effects" introduced by compression in Online Social Networks (OSNs)<n>We propose PLADA, a novel framework designed to tackle the lack of paired data and the ineffective use of compressed images.<n>PLADA consists of two core modules: Block Effect Eraser (B2E), which uses a dual-stage attention mechanism to handle block effects, and Open Data Aggregation (ODA), which processes both paired and unpaired data to improve detection.
  arXiv  Detail & Related papers  (2025-06-25T15:46:41Z)
• Deepfake Media Generation and Detection in the Generative AI Era: A Survey and Outlook [101.30779332427217]
  We survey deepfake generation and detection techniques, including the most recent developments in the field.<n>We identify various kinds of deepfakes, according to the procedure used to alter or generate the fake content.<n>We develop a novel multimodal benchmark to evaluate deepfake detectors on out-of-distribution content.
  arXiv  Detail & Related papers  (2024-11-29T08:29:25Z)
• DF40: Toward Next-Generation Deepfake Detection [62.073997142001424]
  existing works identify top-notch detection algorithms and models by adhering to the common practice: training detectors on one specific dataset and testing them on other prevalent deepfake datasets. But can these stand-out "winners" be truly applied to tackle the myriad of realistic and diverse deepfakes lurking in the real world? We construct a highly diverse deepfake detection dataset called DF40, which comprises 40 distinct deepfake techniques.
  arXiv  Detail & Related papers  (2024-06-19T12:35:02Z)
• AntifakePrompt: Prompt-Tuned Vision-Language Models are Fake Image Detectors [24.78672820633581]
  Deep generative models can create remarkably fake images while raising concerns about misinformation and copyright infringement. Deepfake detection technique is developed to distinguish between real and fake images. We propose a novel approach called AntifakePrompt, using Vision-Language Models and prompt tuning techniques.
  arXiv  Detail & Related papers  (2023-10-26T14:23:45Z)
• Robustness and Generalizability of Deepfake Detection: A Study with Diffusion Models [35.188364409869465]
  We present an investigation into how deepfakes are produced and how they can be identified. The cornerstone of our research is a rich collection of artificial celebrity faces, titled DeepFakeFace. This data serves as a robust foundation to train and test algorithms designed to spot deepfakes.
  arXiv  Detail & Related papers  (2023-09-05T13:22:41Z)
• Fooling State-of-the-Art Deepfake Detection with High-Quality Deepfakes [2.0883760606514934]
  We show that deepfake detectors proven to generalize well on multiple research datasets still struggle in real-world scenarios with well-crafted fakes. We propose a novel autoencoder for face swapping alongside an advanced face blending technique, which we utilize to generate 90 high-quality deepfakes.
  arXiv  Detail & Related papers  (2023-05-09T09:08:49Z)
• Voice-Face Homogeneity Tells Deepfake [56.334968246631725]
  Existing detection approaches contribute to exploring the specific artifacts in deepfake videos. We propose to perform the deepfake detection from an unexplored voice-face matching view. Our model obtains significantly improved performance as compared to other state-of-the-art competitors.
  arXiv  Detail & Related papers  (2022-03-04T09:08:50Z)
• TAR: Generalized Forensic Framework to Detect Deepfakes using Weakly Supervised Learning [17.40885531847159]
  Deepfakes have become a critical social problem, and detecting them is of utmost importance. In this work, we introduce a practical digital forensic tool to detect different types of deepfakes simultaneously. We develop an autoencoder-based detection model with Residual blocks and sequentially perform transfer learning to detect different types of deepfakes simultaneously.
  arXiv  Detail & Related papers  (2021-05-13T07:31:08Z)
• M2TR: Multi-modal Multi-scale Transformers for Deepfake Detection [74.19291916812921]
  forged images generated by Deepfake techniques pose a serious threat to the trustworthiness of digital information. In this paper, we aim to capture the subtle manipulation artifacts at different scales for Deepfake detection. We introduce a high-quality Deepfake dataset, SR-DF, which consists of 4,000 DeepFake videos generated by state-of-the-art face swapping and facial reenactment methods.
  arXiv  Detail & Related papers  (2021-04-20T05:43:44Z)
• WildDeepfake: A Challenging Real-World Dataset for Deepfake Detection [82.42495493102805]
  We introduce a new dataset WildDeepfake which consists of 7,314 face sequences extracted from 707 deepfake videos collected completely from the internet. We conduct a systematic evaluation of a set of baseline detection networks on both existing and our WildDeepfake datasets, and show that WildDeepfake is indeed a more challenging dataset, where the detection performance can decrease drastically.
  arXiv  Detail & Related papers  (2021-01-05T11:10:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.