Adversarial Robustness of AI-Generated Image Detectors in the Real World

• URL: http://arxiv.org/abs/2410.01574v3
• Date: Tue, 03 Jun 2025 16:40:40 GMT
• Title: Adversarial Robustness of AI-Generated Image Detectors in the Real World
• Authors: Sina Mavali, Jonas Ricker, David Pape, Asja Fischer, Lea Schönherr,
• Abstract summary: We show that current state-of-the-art classifiers are vulnerable to adversarial examples under real-world conditions.<n>Most attacks remain effective even when images are degraded during the upload to, e.g., social media platforms.<n>In a case study, we demonstrate that these robustness challenges are also found in commercial tools by conducting black-box attacks on HIVE.
• Score: 13.52355280061187
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The rapid advancement of Generative Artificial Intelligence (GenAI) capabilities is accompanied by a concerning rise in its misuse. In particular the generation of credible misinformation in the form of images poses a significant threat to the public trust in democratic processes. Consequently, there is an urgent need to develop tools to reliably distinguish between authentic and AI-generated content. The majority of detection methods are based on neural networks that are trained to recognize forensic artifacts. In this work, we demonstrate that current state-of-the-art classifiers are vulnerable to adversarial examples under real-world conditions. Through extensive experiments, comprising four detection methods and five attack algorithms, we show that an attacker can dramatically decrease classification performance, without internal knowledge of the detector's architecture. Notably, most attacks remain effective even when images are degraded during the upload to, e.g., social media platforms. In a case study, we demonstrate that these robustness challenges are also found in commercial tools by conducting black-box attacks on HIVE, a proprietary online GenAI media detector. In addition, we evaluate the robustness of using generated features of a robust pre-trained model and showed that this increases the robustness, while not reaching the performance on benign inputs. These results, along with the increasing potential of GenAI to erode public trust, underscore the need for more research and new perspectives on methods to prevent its misuse.

Related papers

• RAID: A Dataset for Testing the Adversarial Robustness of AI-Generated Image Detectors [57.81012948133832]
  We present RAID (Robust evaluation of AI-generated image Detectors), a dataset of 72k diverse and highly transferable adversarial examples.<n>Our methodology generates adversarial images that transfer with a high success rate to unseen detectors.<n>Our findings indicate that current state-of-the-art AI-generated image detectors can be easily deceived by adversarial examples.
  arXiv  Detail & Related papers  (2025-06-04T14:16:00Z)
• Adversarially Robust AI-Generated Image Detection for Free: An Information Theoretic Perspective [22.514709685678813]
  We show that adversarial training (AT) suffers from performance collapse in AIGI detection.<n>Motivated by this difference, we propose Training-free Robust Detection via Information-theoretic Measures (TRIM)<n>TRIM builds on standard detectors and quantifies feature shifts using prediction entropy and KL divergence.
  arXiv  Detail & Related papers  (2025-05-28T17:20:49Z)
• Robustness in AI-Generated Detection: Enhancing Resistance to Adversarial Attacks [4.179092469766417]
  This paper investigates the vulnerabilities of current AI-generated face detection systems.<n>We propose an approach that integrates adversarial training to mitigate the impact of adversarial examples.<n>We also provide an in-depth analysis of adversarial and benign examples, offering insights into the intrinsic characteristics of AI-generated content.
  arXiv  Detail & Related papers  (2025-05-06T11:19:01Z)
• Computational Safety for Generative AI: A Signal Processing Perspective [65.268245109828]
  computational safety is a mathematical framework that enables the quantitative assessment, formulation, and study of safety challenges in GenAI.<n>We show how sensitivity analysis and loss landscape analysis can be used to detect malicious prompts with jailbreak attempts.<n>We discuss key open research challenges, opportunities, and the essential role of signal processing in computational AI safety.
  arXiv  Detail & Related papers  (2025-02-18T02:26:50Z)
• Time-Aware Face Anti-Spoofing with Rotation Invariant Local Binary Patterns and Deep Learning [50.79277723970418]
  imitation attacks can lead to erroneous identification and subsequent authentication of attackers. Similar to face recognition, imitation attacks can also be detected with Machine Learning. We propose a novel approach that promises high classification accuracy by combining previously unused features with time-aware deep learning strategies.
  arXiv  Detail & Related papers  (2024-08-27T07:26:10Z)
• Vulnerabilities in AI-generated Image Detection: The Challenge of Adversarial Attacks [39.524974831780874]
  We show that adversarial attack is truly a real threat to AIGI detectors, because FPBA can deliver successful black-box attacks. We name our method as Frequency-based Post-train Bayesian Attack, or FPBA.
  arXiv  Detail & Related papers  (2024-07-30T14:07:17Z)
• Imperceptible Face Forgery Attack via Adversarial Semantic Mask [59.23247545399068]
  We propose an Adversarial Semantic Mask Attack framework (ASMA) which can generate adversarial examples with good transferability and invisibility. Specifically, we propose a novel adversarial semantic mask generative model, which can constrain generated perturbations in local semantic regions for good stealthiness.
  arXiv  Detail & Related papers  (2024-06-16T10:38:11Z)
• Humanizing Machine-Generated Content: Evading AI-Text Detection through Adversarial Attack [24.954755569786396]
  We propose a framework for a broader class of adversarial attacks, designed to perform minor perturbations in machine-generated content to evade detection. We consider two attack settings: white-box and black-box, and employ adversarial learning in dynamic scenarios to assess the potential enhancement of the current detection model's robustness. The empirical results reveal that the current detection models can be compromised in as little as 10 seconds, leading to the misclassification of machine-generated text as human-written content.
  arXiv  Detail & Related papers  (2024-04-02T12:49:22Z)
• Deepfake Sentry: Harnessing Ensemble Intelligence for Resilient Detection and Generalisation [0.8796261172196743]
  We propose a proactive and sustainable deepfake training augmentation solution. We employ a pool of autoencoders that mimic the effect of the artefacts introduced by the deepfake generator models. Experiments reveal that our proposed ensemble autoencoder-based data augmentation learning approach offers improvements in terms of generalisation.
  arXiv  Detail & Related papers  (2024-03-29T19:09:08Z)
• Real is not True: Backdoor Attacks Against Deepfake Detection [9.572726483706846]
  We introduce a pioneering paradigm denominated as Bad-Deepfake, which represents a novel foray into the realm of backdoor attacks levied against deepfake detectors. Our approach hinges upon the strategic manipulation of a subset of the training data, enabling us to wield disproportionate influence over the operational characteristics of a trained model.
  arXiv  Detail & Related papers  (2024-03-11T10:57:14Z)
• XAI-Based Detection of Adversarial Attacks on Deepfake Detectors [0.0]
  We introduce a novel methodology for identifying adversarial attacks on deepfake detectors using XAI. Our approach contributes not only to the detection of deepfakes but also enhances the understanding of possible adversarial attacks.
  arXiv  Detail & Related papers  (2024-03-05T13:25:30Z)
• Evading Forensic Classifiers with Attribute-Conditioned Adversarial Faces [6.105361899083232]
  We show that it is possible to successfully generate adversarial fake faces with a specified set of attributes. We propose a framework to search for adversarial latent codes within the feature space of StyleGAN. We also propose a meta-learning based optimization strategy to achieve transferable performance on unknown target models.
  arXiv  Detail & Related papers  (2023-06-22T17:59:55Z)
• Can AI-Generated Text be Reliably Detected? [54.670136179857344]
  Unregulated use of LLMs can potentially lead to malicious consequences such as plagiarism, generating fake news, spamming, etc. Recent works attempt to tackle this problem either using certain model signatures present in the generated text outputs or by applying watermarking techniques. In this paper, we show that these detectors are not reliable in practical scenarios.
  arXiv  Detail & Related papers  (2023-03-17T17:53:19Z)
• Illusory Attacks: Information-Theoretic Detectability Matters in Adversarial Attacks [76.35478518372692]
  We introduce epsilon-illusory, a novel form of adversarial attack on sequential decision-makers. Compared to existing attacks, we empirically find epsilon-illusory to be significantly harder to detect with automated methods. Our findings suggest the need for better anomaly detectors, as well as effective hardware- and system-level defenses.
  arXiv  Detail & Related papers  (2022-07-20T19:49:09Z)
• Adversarially-Aware Robust Object Detector [85.10894272034135]
  We propose a Robust Detector (RobustDet) based on adversarially-aware convolution to disentangle gradients for model learning on clean and adversarial images. Our model effectively disentangles gradients and significantly enhances the detection robustness with maintaining the detection ability on clean images.
  arXiv  Detail & Related papers  (2022-07-13T13:59:59Z)
• Real-World Adversarial Examples involving Makeup Application [58.731070632586594]
  We propose a physical adversarial attack with the use of full-face makeup. Our attack can effectively overcome manual errors in makeup application, such as color and position-related errors.
  arXiv  Detail & Related papers  (2021-09-04T05:29:28Z)
• Deepfake Forensics via An Adversarial Game [99.84099103679816]
  We advocate adversarial training for improving the generalization ability to both unseen facial forgeries and unseen image/video qualities. Considering that AI-based face manipulation often leads to high-frequency artifacts that can be easily spotted by models yet difficult to generalize, we propose a new adversarial training method that attempts to blur out these specific artifacts.
  arXiv  Detail & Related papers  (2021-03-25T02:20:08Z)
• Adversarial Attacks for Tabular Data: Application to Fraud Detection and Imbalanced Data [3.2458203725405976]
  Adversarial attacks aim at producing adversarial examples, in other words, slightly modified inputs that induce the AI system to return incorrect outputs. In this paper we illustrate a novel approach to modify and adapt state-of-the-art algorithms to imbalanced data, in the context of fraud detection. Experimental results show that the proposed modifications lead to a perfect attack success rate. When applied to a real-world production system, the proposed techniques shows the possibility of posing a serious threat to the robustness of advanced AI-based fraud detection procedures.
  arXiv  Detail & Related papers  (2021-01-20T08:58:29Z)
• Artificial Fingerprinting for Generative Models: Rooting Deepfake Attribution in Training Data [64.65952078807086]
  Photorealistic image generation has reached a new level of quality due to the breakthroughs of generative adversarial networks (GANs) Yet, the dark side of such deepfakes, the malicious use of generated media, raises concerns about visual misinformation. We seek a proactive and sustainable solution on deepfake detection by introducing artificial fingerprints into the models.
  arXiv  Detail & Related papers  (2020-07-16T16:49:55Z)
• Temporal Sparse Adversarial Attack on Sequence-based Gait Recognition [56.844587127848854]
  We demonstrate that the state-of-the-art gait recognition model is vulnerable to such attacks. We employ a generative adversarial network based architecture to semantically generate adversarial high-quality gait silhouettes or video frames. The experimental results show that if only one-fortieth of the frames are attacked, the accuracy of the target model drops dramatically.
  arXiv  Detail & Related papers  (2020-02-22T10:08:42Z)
• Adversarial vs behavioural-based defensive AI with joint, continual and active learning: automated evaluation of robustness to deception, poisoning and concept drift [62.997667081978825]
  Recent advancements in Artificial Intelligence (AI) have brought new capabilities to behavioural analysis (UEBA) for cyber-security. In this paper, we present a solution to effectively mitigate this attack by improving the detection process and efficiently leveraging human expertise.
  arXiv  Detail & Related papers  (2020-01-13T13:54:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.