Deepfake-Eval-2024: A Multi-Modal In-the-Wild Benchmark of Deepfakes Circulated in 2024

• URL: http://arxiv.org/abs/2503.02857v4
• Date: Tue, 27 May 2025 16:27:56 GMT
• Title: Deepfake-Eval-2024: A Multi-Modal In-the-Wild Benchmark of Deepfakes Circulated in 2024
• Abstract summary: We introduce Deepfake-Eval-2024, a new deepfake detection benchmark consisting of in-the-wild deepfakes collected from social media and deepfake detection platform users in 2024.<n>The benchmark contains diverse media content from 88 different websites in 52 different languages.<n>We find that the performance of open-source state-of-the-art deepfake detection models drops precipitously when evaluated on Deepfake-Eval-2024.
• Score: 6.865766644176186
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: In the age of increasingly realistic generative AI, robust deepfake detection is essential for mitigating fraud and disinformation. While many deepfake detectors report high accuracy on academic datasets, we show that these academic benchmarks are out of date and not representative of real-world deepfakes. We introduce Deepfake-Eval-2024, a new deepfake detection benchmark consisting of in-the-wild deepfakes collected from social media and deepfake detection platform users in 2024. Deepfake-Eval-2024 consists of 45 hours of videos, 56.5 hours of audio, and 1,975 images, encompassing the latest manipulation technologies. The benchmark contains diverse media content from 88 different websites in 52 different languages. We find that the performance of open-source state-of-the-art deepfake detection models drops precipitously when evaluated on Deepfake-Eval-2024, with AUC decreasing by 50% for video, 48% for audio, and 45% for image models compared to previous benchmarks. We also evaluate commercial deepfake detection models and models finetuned on Deepfake-Eval-2024, and find that they have superior performance to off-the-shelf open-source models, but do not yet reach the accuracy of deepfake forensic analysts. The dataset is available at https://github.com/nuriachandra/Deepfake-Eval-2024.

Related papers

• Revisiting Simple Baselines for In-The-Wild Deepfake Detection [0.0]
  We focus on the recently released "in-the-wild" benchmark, Deepfake-Eval-2024.<n>Three finetuned open-source models achieve accuracies between 61% and 69%, significantly lagging behind the leading commercial deepfake detector with 82% accuracy.<n>Our work revisits one of these baseline approaches, which adapts standard pretrained vision backbones to produce generalizable deepfake detectors.
  arXiv  Detail & Related papers  (2025-09-04T12:23:59Z)
• Evaluating Deepfake Detectors in the Wild [0.0]
  We evaluate modern deepfake detectors, introducing a novel testing procedure designed to mimic real-world scenarios for deepfake detection.<n>Our analysis shows that detecting deepfakes still remains a challenging task.<n>Basic image manipulations, such as JPEG compression or image enhancement, can significantly reduce model performance.
  arXiv  Detail & Related papers  (2025-07-29T15:17:00Z)
• Unmasking Deep Fakes: Leveraging Deep Learning for Video Authenticity Detection [3.483595743063401]
  The primary motivation of this paper is to recognize deepfake videos using deep learning techniques.<n>We consider using MTCNN as a face detector and EfficientNet-B5 as encoder model to predict if a video is deepfake or not.<n>The results show that our deepfake detection model acquired 42.78% log loss, 93.80% AUC and 86.82% F1 score on kaggle's DFDC dataset.
  arXiv  Detail & Related papers  (2025-05-10T06:19:14Z)
• Deepfake Media Generation and Detection in the Generative AI Era: A Survey and Outlook [101.30779332427217]
  We survey deepfake generation and detection techniques, including the most recent developments in the field.<n>We identify various kinds of deepfakes, according to the procedure used to alter or generate the fake content.<n>We develop a novel multimodal benchmark to evaluate deepfake detectors on out-of-distribution content.
  arXiv  Detail & Related papers  (2024-11-29T08:29:25Z)
• DF40: Toward Next-Generation Deepfake Detection [62.073997142001424]
  existing works identify top-notch detection algorithms and models by adhering to the common practice: training detectors on one specific dataset and testing them on other prevalent deepfake datasets. But can these stand-out "winners" be truly applied to tackle the myriad of realistic and diverse deepfakes lurking in the real world? We construct a highly diverse deepfake detection dataset called DF40, which comprises 40 distinct deepfake techniques.
  arXiv  Detail & Related papers  (2024-06-19T12:35:02Z)
• Leveraging Deep Learning Approaches for Deepfake Detection: A Review [0.0]
  Deepfakes are fabricated media generated by AI that are difficult to set apart from the real media. This paper aims to explore different methodologies with an intention to achieve a cost-effective model.
  arXiv  Detail & Related papers  (2023-04-04T16:04:42Z)
• DeePhy: On Deepfake Phylogeny [58.01631614114075]
  DeePhy is a novel Deepfake Phylogeny dataset which consists of 5040 deepfake videos generated using three different generation techniques. We present the benchmark on DeePhy dataset using six deepfake detection algorithms.
  arXiv  Detail & Related papers  (2022-09-19T15:30:33Z)
• A Continual Deepfake Detection Benchmark: Dataset, Methods, and Essentials [97.69553832500547]
  This paper suggests a continual deepfake detection benchmark (CDDB) over a new collection of deepfakes from both known and unknown generative models. We exploit multiple approaches to adapt multiclass incremental learning methods, commonly used in the continual visual recognition, to the continual deepfake detection problem.
  arXiv  Detail & Related papers  (2022-05-11T13:07:19Z)
• Model Attribution of Face-swap Deepfake Videos [39.771800841412414]
  We first introduce a new dataset with DeepFakes from Different Models (DFDM) based on several Autoencoder models. Specifically, five generation models with variations in encoder, decoder, intermediate layer, input resolution, and compression ratio have been used to generate a total of 6,450 Deepfake videos. We take Deepfakes model attribution as a multiclass classification task and propose a spatial and temporal attention based method to explore the differences among Deepfakes.
  arXiv  Detail & Related papers  (2022-02-25T20:05:18Z)
• An Experimental Evaluation on Deepfake Detection using Deep Face Recognition [0.0]
  Deep learning has led to the generation of very realistic fake content, also known as deepfakes. Most of the current deepfake detection methods are deemed as a binary classification problem in distinguishing authentic images or videos from fake ones using two-class convolutional neural networks (CNNs) This paper thoroughly evaluate the efficacy of deep face recognition in identifying deepfakes, using different loss functions and deepfake generation techniques.
  arXiv  Detail & Related papers  (2021-10-04T18:02:56Z)
• TAR: Generalized Forensic Framework to Detect Deepfakes using Weakly Supervised Learning [17.40885531847159]
  Deepfakes have become a critical social problem, and detecting them is of utmost importance. In this work, we introduce a practical digital forensic tool to detect different types of deepfakes simultaneously. We develop an autoencoder-based detection model with Residual blocks and sequentially perform transfer learning to detect different types of deepfakes simultaneously.
  arXiv  Detail & Related papers  (2021-05-13T07:31:08Z)
• Adversarially robust deepfake media detection using fused convolutional neural network predictions [79.00202519223662]
  Current deepfake detection systems struggle against unseen data. We employ three different deep Convolutional Neural Network (CNN) models to classify fake and real images extracted from videos. The proposed technique outperforms state-of-the-art models with 96.5% accuracy.
  arXiv  Detail & Related papers  (2021-02-11T11:28:00Z)
• WildDeepfake: A Challenging Real-World Dataset for Deepfake Detection [82.42495493102805]
  We introduce a new dataset WildDeepfake which consists of 7,314 face sequences extracted from 707 deepfake videos collected completely from the internet. We conduct a systematic evaluation of a set of baseline detection networks on both existing and our WildDeepfake datasets, and show that WildDeepfake is indeed a more challenging dataset, where the detection performance can decrease drastically.
  arXiv  Detail & Related papers  (2021-01-05T11:10:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.