AI-generated Images Challenge Visual Trust in High-risk Scenarios

• URL: http://arxiv.org/abs/2607.22745v1
• Date: Thu, 23 Jul 2026 06:56:49 GMT
• Title: AI-generated Images Challenge Visual Trust in High-risk Scenarios
• Abstract summary: We introduce SafeIMG, a safety-oriented benchmark spanning 12 public- and individual-safety scenarios generated using GPT Image 2.<n>SafeIMG evaluates whether detectors recognise synthetic images, but also whether their decisions reflect human-identified anomalies.<n>We evaluate specialized synthetic-image detectors and vision-language models (VLMs), and find that neither provides reliable detection.
• Score: 79.40239296310791
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Rapid advances in image generation are eroding the evidentiary value of visual content in settings where authenticity can affect public safety and personal reputation. Yet existing detection benchmarks rarely examine synthetic images in public- and individual-safety contexts, where misleading visual content may carry substantial risks. Here we introduce SafeIMG, a safety-oriented benchmark spanning 12 public- and individual-safety scenarios generated using GPT Image 2. Unlike benchmarks centred on generic imagery and image-level labels, SafeIMG evaluates not only whether detectors recognise synthetic images, but also whether their decisions reflect human-identified anomalies. To this end, SafeIMG provides human annotations that localise suspicious regions and explain local artefacts and higher-level commonsense or physical inconsistencies. We evaluate specialized synthetic-image detectors and vision-language models (VLMs), and find that neither provides reliable detection. The strongest VLM identifies only 49.5% of generated images, whereas the best specialised detector identifies 33.1%, compared with 81.7% accuracy for human evaluators. Model explanations cover only 29.8\% of human-annotated anomalies and predominantly capture local defects in text, faces and hands. Their coverage falls to 15.0% for commonsense conflicts and 12.0% for physical inconsistencies, while detection performance deteriorates further after dissemination-induced image degradation. These findings show that current detectors lack the accuracy, explanatory alignment and robustness needed to evaluate AI-generated images reliably across public- and individual-safety settings.

Related papers

• Text Over Image: Auditing Multimodal Robustness in Synthetic Medical Image Detection [38.146353416394234]
  We reformulate synthetic medical image detection as an audit of multimodal robustness at the image-record interface.<n>We find that changing the metadata context alone can flip authenticity judgments.<n>We propose an inference-time mitigation pipeline that detects and neutralizes provenance shortcuts without model retraining.
  arXiv  Detail & Related papers  (2026-06-24T04:08:41Z)
• Forged Calamity: Benchmark for Cross-Domain Synthetic Disaster Detection in the Age of Diffusion [20.58307772592017]
  We introduce Forged Calamity, a benchmark dataset for synthetic disaster detection containing 30,000 images.<n>Experiments across fine-tuned and zero-shot settings reveal consistent weaknesses in current forensic approaches.<n>These findings highlight persistent generalization gaps and the urgent need for domain- and model-agnostic detection methods.
  arXiv  Detail & Related papers  (2026-06-17T00:08:35Z)
• VLSU: Mapping the Limits of Joint Multimodal Understanding for AI Safety [3.1109025622085693]
  We present Vision Language Safety Understanding, a comprehensive framework to evaluate multimodal safety.<n>Our evaluation of eleven state-of-the-art models reveals systematic joint understanding failures.<n>Our framework exposes weaknesses in joint image-text understanding and alignment gaps in current models.
  arXiv  Detail & Related papers  (2025-10-21T01:30:31Z)
• Bridging the Gap Between Ideal and Real-world Evaluation: Benchmarking AI-Generated Image Detection in Challenging Scenarios [54.07895223545793]
  This paper introduces the Real-World Robustness dataset (RRDataset) for comprehensive evaluation of detection models across three dimensions.<n>RRDataset includes high-quality images from seven major scenarios.<n>We benchmarked 17 detectors and 10 vision-language models (VLMs) on RRDataset and conducted a large-scale human study.
  arXiv  Detail & Related papers  (2025-09-11T06:15:52Z)
• RAID: A Dataset for Testing the Adversarial Robustness of AI-Generated Image Detectors [57.81012948133832]
  We present RAID (Robust evaluation of AI-generated image Detectors), a dataset of 72k diverse and highly transferable adversarial examples.<n>Our methodology generates adversarial images that transfer with a high success rate to unseen detectors.<n>Our findings indicate that current state-of-the-art AI-generated image detectors can be easily deceived by adversarial examples.
  arXiv  Detail & Related papers  (2025-06-04T14:16:00Z)
• Adversarially Robust AI-Generated Image Detection for Free: An Information Theoretic Perspective [22.514709685678813]
  We show that adversarial training (AT) suffers from performance collapse in AIGI detection.<n>Motivated by this difference, we propose Training-free Robust Detection via Information-theoretic Measures (TRIM)<n>TRIM builds on standard detectors and quantifies feature shifts using prediction entropy and KL divergence.
  arXiv  Detail & Related papers  (2025-05-28T17:20:49Z)
• Better Safe Than Sorry? Overreaction Problem of Vision Language Models in Visual Emergency Recognition [12.054081112688074]
  Vision-Language Models (VLMs) have shown capabilities in interpreting visual content, but their reliability in safety-critical scenarios remains insufficiently explored.<n>We introduce VERI, a diagnostic benchmark comprising 200 synthetic images (100 contrastive pairs) and an additional 50 real-world images (25 pairs) for validation.<n>Each emergency scene is paired with a visually similar but safe counterpart through human verification.
  arXiv  Detail & Related papers  (2025-05-21T10:57:40Z)
• Epistemic Uncertainty for Generated Image Detection [107.62647907393377]
  We introduce a novel framework for AI-generated image detection through epistemic uncertainty, aiming to address critical security concerns in the era of generative models.<n>Our key insight stems from the observation that distributional discrepancies between training and testing data manifest distinctively in the epistemic uncertainty space of machine learning models.
  arXiv  Detail & Related papers  (2024-12-08T11:32:25Z)
• Semi-Truths: A Large-Scale Dataset of AI-Augmented Images for Evaluating Robustness of AI-Generated Image detectors [62.63467652611788]
  We introduce SEMI-TRUTHS, featuring 27,600 real images, 223,400 masks, and 1,472,700 AI-augmented images. Each augmented image is accompanied by metadata for standardized and targeted evaluation of detector robustness. Our findings suggest that state-of-the-art detectors exhibit varying sensitivities to the types and degrees of perturbations, data distributions, and augmentation methods used.
  arXiv  Detail & Related papers  (2024-11-12T01:17:27Z)
• Diffusion Facial Forgery Detection [56.69763252655695]
  This paper introduces DiFF, a comprehensive dataset dedicated to face-focused diffusion-generated images. We conduct extensive experiments on the DiFF dataset via a human test and several representative forgery detection methods. The results demonstrate that the binary detection accuracy of both human observers and automated detectors often falls below 30%.
  arXiv  Detail & Related papers  (2024-01-29T03:20:19Z)
• ArtiFact: A Large-Scale Dataset with Artificial and Factual Images for Generalizable and Robust Synthetic Image Detection [0.3779860024918729]
  This paper assesses the generalizability and robustness of synthetic image detectors in the face of real-world impairments. A proposed multi-class classification scheme, combined with a filter stride reduction strategy addresses social platform impairments. The solution significantly outperforms other top teams by 8.34% on Test 1, 1.26% on Test 2, and 15.08% on Test 3 in the IEEE VIP Cup challenge at ICIP 2022, as measured by the accuracy metric.
  arXiv  Detail & Related papers  (2023-02-23T12:40:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.