What Do Deepfake Benchmarks Measure? An Audit Using Frozen Self-Supervised Representations

• URL: http://arxiv.org/abs/2606.26384v1
• Date: Wed, 24 Jun 2026 21:10:42 GMT
• Title: What Do Deepfake Benchmarks Measure? An Audit Using Frozen Self-Supervised Representations
• Abstract summary: A concerning feedback loop has emerged: benchmarks drive increasingly complex, engineered detectors, yet if those benchmarks do not reflect real-world deepfakes, this complexity may be solving the wrong problem entirely.<n>We conduct an audit of video, image, and audio deepfake benchmarks using a deliberately simple diagnostic.<n>We observe, across three modalities, linear probes on general-purpose self-supervised representations closely approach the performance of bespoke detectors.
• Score: 9.75344282356567
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: As deepfake generators approach perceptual indistinguishability, reliable detection becomes critical. Yet, detectors that score well on benchmarks routinely fail in the wild. A concerning feedback loop has emerged: benchmarks drive increasingly complex, engineered detectors, yet if those benchmarks do not reflect real-world deepfakes, this complexity may be solving the wrong problem entirely. This raises a prior question: what are these benchmarks actually measuring? We conduct an audit of video, image, and audio deepfake benchmarks using a deliberately simple diagnostic. If a linear probe on frozen, general-purpose self-supervised representations can approximate the performance of a bespoke detector, the benchmark is largely rewarding general modality understanding rather than forensic understanding. This has two implications: the benchmark may not reflect realistic threat models, and it raises the question of whether the bespoke detectors the probe approaches are truly learning forensic understanding. We observe, across three modalities, linear probes on general-purpose self-supervised representations closely approach the performance of bespoke detectors. We further show that generator-level difficulty is partly explained by Frechet geometry in the same representation space. Together, these results support a benchmark-audit view of deepfake detection: before high scores are read as evidence of forensic understanding, it is worth asking how much of the benchmark is already solved by general-purpose representations.

Related papers

• Automated Transcript Analysis for Detecting Flaws in Agentic Benchmarks [2.4035041353107673]
  We develop AI scanners to detect four types of validity issues: ground truth access, tool failure, guessing vulnerability, and answer format ambiguity.<n>Our scanners identify several verified quality issues in five widely used benchmarks, including cases unlikely to be caught by random manual inspection.
  arXiv  Detail & Related papers  (2026-07-29T23:17:08Z)
• PARALLAX: Separating Genuine Hallucination Detection from Benchmark Construction Artifacts [12.088141798203367]
  Large language models (LLMs) hallucinate with confidence.<n> detecting it from internal model states offers a path to safer deployment.<n>Recent methods achieving high detection performance on widely used benchmarks.<n>We show that much of this apparent progress does not survive scrutiny.
  arXiv  Detail & Related papers  (2026-05-16T14:57:15Z)
• Beyond Raw Detection Scores: Markov-Informed Calibration for Boosting Machine-Generated Text Detection [105.14032334647932]
  Machine-generated texts (MGTs) pose risks such as disinformation and phishing, highlighting the need for reliable detection.<n> Metric-based methods, which extract statistically distinguishable features of MGTs, are often more practical than complex model-based methods that are prone to overfitting.<n>We propose a Markov-informed score calibration strategy that models two relationships of context detection scores that may aid calibration.
  arXiv  Detail & Related papers  (2026-02-08T16:06:12Z)
• Advancing Machine-Generated Text Detection from an Easy to Hard Supervision Perspective [108.30620357325559]
  Existing machine-generated text (MGT) detection methods implicitly assume labels as the "golden standard"<n>We propose an easy-to-hard enhancement framework to provide reliable supervision under such inexact conditions.
  arXiv  Detail & Related papers  (2025-11-02T15:59:31Z)
• Why Speech Deepfake Detectors Won't Generalize: The Limits of Detection in an Open World [11.238970239267248]
  Speech deepfake detectors are often evaluated on clean, benchmark-style conditions.<n>But deployment occurs in an open world of shifting devices, sampling rates, codecs, environments, and attack families.<n>This creates a coverage debt" for AI-based detectors, producing data blind spots that grow faster than data can be collected.
  arXiv  Detail & Related papers  (2025-09-23T20:27:04Z)
• Do Large Language Model Benchmarks Test Reliability? [66.1783478365998]
  We investigate how well current benchmarks quantify model reliability.<n>Motivated by this gap in the evaluation of reliability, we propose the concept of so-called platinum benchmarks.<n>We evaluate a wide range of models on these platinum benchmarks and find that, indeed, frontier LLMs still exhibit failures on simple tasks.
  arXiv  Detail & Related papers  (2025-02-05T18:58:19Z)
• OoDIS: Anomaly Instance Segmentation and Detection Benchmark [57.89836988990543]
  This work extends some commonly used anomaly segmentation benchmarks to include the instance segmentation and object detection tasks.<n>Our evaluation of anomaly segmentation and object detection methods shows that both of these challenges remain unsolved problems.
  arXiv  Detail & Related papers  (2024-06-17T17:59:56Z)
• DeepfakeBench: A Comprehensive Benchmark of Deepfake Detection [55.70982767084996]
  A critical yet frequently overlooked challenge in the field of deepfake detection is the lack of a standardized, unified, comprehensive benchmark. We present the first comprehensive benchmark for deepfake detection, called DeepfakeBench, which offers three key contributions. DeepfakeBench contains 15 state-of-the-art detection methods, 9CL datasets, a series of deepfake detection evaluation protocols and analysis tools, as well as comprehensive evaluations.
  arXiv  Detail & Related papers  (2023-07-04T01:34:41Z)
• Towards Computational Performance Engineering for Unsupervised Concept Drift Detection -- Complexities, Benchmarking, Performance Analysis [4.720921955899519]
  Concept drift detection is crucial for many AI systems to ensure the system's reliability. These systems often have to deal with large amounts of data or react in real-time. drift detectors must meet computational requirements or constraints with a comprehensive performance evaluation.
  arXiv  Detail & Related papers  (2023-04-17T14:39:56Z)
• Pedestrian Detection: Domain Generalization, CNNs, Transformers and Beyond [82.37430109152383]
  We show that, current pedestrian detectors poorly handle even small domain shifts in cross-dataset evaluation. We attribute the limited generalization to two main factors, the method and the current sources of data. We propose a progressive fine-tuning strategy which improves generalization.
  arXiv  Detail & Related papers  (2022-01-10T06:00:26Z)
• Impact of Benign Modifications on Discriminative Performance of Deepfake Detectors [11.881119750753648]
  A large number of deepfake detectors have been proposed recently in order to identify such content. Deepfakes are increasingly popular in both good faith applications such as in entertainment and maliciously intended manipulations such as in image and video forgery. This paper proposes a more rigorous and systematic framework to assess the performance of deepfake detectors in more realistic situations.
  arXiv  Detail & Related papers  (2021-11-14T22:50:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.