Forensic Self-Descriptions Are All You Need for Zero-Shot Detection, Open-Set Source Attribution, and Clustering of AI-generated Images

• URL: http://arxiv.org/abs/2503.21003v1
• Date: Wed, 26 Mar 2025 21:34:37 GMT
• Title: Forensic Self-Descriptions Are All You Need for Zero-Shot Detection, Open-Set Source Attribution, and Clustering of AI-generated Images
• Authors: Tai D. Nguyen, Aref Azizpour, Matthew C. Stamm,
• Abstract summary: Traditional methods fail to generalize to unseen generators due to reliance on features specific to known sources during training.<n>We propose a novel approach that explicitly models forensic microstructures.<n>This self-description enables us to perform zero-shot detection of synthetic images, open-set source attribution of images, and clustering based on source without prior knowledge.
• Score: 8.167678851224121
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: The emergence of advanced AI-based tools to generate realistic images poses significant challenges for forensic detection and source attribution, especially as new generative techniques appear rapidly. Traditional methods often fail to generalize to unseen generators due to reliance on features specific to known sources during training. To address this problem, we propose a novel approach that explicitly models forensic microstructures - subtle, pixel-level patterns unique to the image creation process. Using only real images in a self-supervised manner, we learn a set of diverse predictive filters to extract residuals that capture different aspects of these microstructures. By jointly modeling these residuals across multiple scales, we obtain a compact model whose parameters constitute a unique forensic self-description for each image. This self-description enables us to perform zero-shot detection of synthetic images, open-set source attribution of images, and clustering based on source without prior knowledge. Extensive experiments demonstrate that our method achieves superior accuracy and adaptability compared to competing techniques, advancing the state of the art in synthetic media forensics.

Related papers

• FakeScope: Large Multimodal Expert Model for Transparent AI-Generated Image Forensics [66.14786900470158]
  We propose FakeScope, an expert multimodal model (LMM) tailored for AI-generated image forensics. FakeScope identifies AI-synthetic images with high accuracy and provides rich, interpretable, and query-driven forensic insights. FakeScope achieves state-of-the-art performance in both closed-ended and open-ended forensic scenarios.
  arXiv  Detail & Related papers  (2025-03-31T16:12:48Z)
• Time Step Generating: A Universal Synthesized Deepfake Image Detector [0.4488895231267077]
  We propose a universal synthetic image detector Time Step Generating (TSG) TSG does not rely on pre-trained models' reconstructing ability, specific datasets, or sampling algorithms. We test the proposed TSG on the large-scale GenImage benchmark and it achieves significant improvements in both accuracy and generalizability.
  arXiv  Detail & Related papers  (2024-11-17T09:39:50Z)
• Are CLIP features all you need for Universal Synthetic Image Origin Attribution? [13.96698277726253]
  We propose a framework that incorporates features from large pre-trained foundation models to perform Open-Set origin attribution of synthetic images. We show that our method leads to remarkable attribution performance, even in the low-data regime.
  arXiv  Detail & Related papers  (2024-08-17T09:54:21Z)
• Contrasting Deepfakes Diffusion via Contrastive Learning and Global-Local Similarities [88.398085358514]
  Contrastive Deepfake Embeddings (CoDE) is a novel embedding space specifically designed for deepfake detection. CoDE is trained via contrastive learning by additionally enforcing global-local similarities.
  arXiv  Detail & Related papers  (2024-07-29T18:00:10Z)
• BOSC: A Backdoor-based Framework for Open Set Synthetic Image Attribution [22.81354665006496]
  Synthetic image attribution addresses the problem of tracing back the origin of images produced by generative models. We propose a framework for open set attribution of synthetic images, named BOSC, that relies on the concept of backdoor attacks.
  arXiv  Detail & Related papers  (2024-05-19T09:17:43Z)
• LCM-Lookahead for Encoder-based Text-to-Image Personalization [82.56471486184252]
  We explore the potential of using shortcut-mechanisms to guide the personalization of text-to-image models. We focus on encoder-based personalization approaches, and demonstrate that by tuning them with a lookahead identity loss, we can achieve higher identity fidelity.
  arXiv  Detail & Related papers  (2024-04-04T17:43:06Z)
• ASAP: Interpretable Analysis and Summarization of AI-generated Image Patterns at Scale [20.12991230544801]
  Generative image models have emerged as a promising technology to produce realistic images. There is growing demand to empower users to effectively discern and comprehend patterns of AI-generated images. We develop ASAP, an interactive visualization system that automatically extracts distinct patterns of AI-generated images.
  arXiv  Detail & Related papers  (2024-04-03T18:20:41Z)
• Perceptual Artifacts Localization for Image Synthesis Tasks [59.638307505334076]
  We introduce a novel dataset comprising 10,168 generated images, each annotated with per-pixel perceptual artifact labels. A segmentation model, trained on our proposed dataset, effectively localizes artifacts across a range of tasks. We propose an innovative zoom-in inpainting pipeline that seamlessly rectifies perceptual artifacts in the generated images.
  arXiv  Detail & Related papers  (2023-10-09T10:22:08Z)
• Generalizable Synthetic Image Detection via Language-guided Contrastive Learning [22.4158195581231]
  malevolent use of synthetic images, such as the dissemination of fake news or the creation of fake profiles, raises significant concerns regarding the authenticity of images. We propose a simple yet very effective synthetic image detection method via a language-guided contrastive learning and a new formulation of the detection problem. It is shown that our proposed LanguAge-guided SynThEsis Detection (LASTED) model achieves much improved generalizability to unseen image generation models.
  arXiv  Detail & Related papers  (2023-05-23T08:13:27Z)
• Parents and Children: Distinguishing Multimodal DeepFakes from Natural Images [60.34381768479834]
  Recent advancements in diffusion models have enabled the generation of realistic deepfakes from textual prompts in natural language. We pioneer a systematic study on deepfake detection generated by state-of-the-art diffusion models.
  arXiv  Detail & Related papers  (2023-04-02T10:25:09Z)
• Self-supervised GAN Detector [10.963740942220168]
  generative models can be abused with malicious purposes, such as fraud, defamation, and fake news. We propose a novel framework to distinguish the unseen generated images outside of the training settings. Our proposed method is composed of the artificial fingerprint generator reconstructing the high-quality artificial fingerprints of GAN images.
  arXiv  Detail & Related papers  (2021-11-12T06:19:04Z)
• SIR: Self-supervised Image Rectification via Seeing the Same Scene from Multiple Different Lenses [82.56853587380168]
  We propose a novel self-supervised image rectification (SIR) method based on an important insight that the rectified results of distorted images of the same scene from different lens should be the same. We leverage a differentiable warping module to generate the rectified images and re-distorted images from the distortion parameters. Our method achieves comparable or even better performance than the supervised baseline method and representative state-of-the-art methods.
  arXiv  Detail & Related papers  (2020-11-30T08:23:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.