Are CLIP features all you need for Universal Synthetic Image Origin Attribution?

• URL: http://arxiv.org/abs/2408.09153v1
• Date: Sat, 17 Aug 2024 09:54:21 GMT
• Title: Are CLIP features all you need for Universal Synthetic Image Origin Attribution?
• Authors: Dario Cioni, Christos Tzelepis, Lorenzo Seidenari, Ioannis Patras,
• Abstract summary: We propose a framework that incorporates features from large pre-trained foundation models to perform Open-Set origin attribution of synthetic images. We show that our method leads to remarkable attribution performance, even in the low-data regime.
• Score: 13.96698277726253
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The steady improvement of Diffusion Models for visual synthesis has given rise to many new and interesting use cases of synthetic images but also has raised concerns about their potential abuse, which poses significant societal threats. To address this, fake images need to be detected and attributed to their source model, and given the frequent release of new generators, realistic applications need to consider an Open-Set scenario where some models are unseen at training time. Existing forensic techniques are either limited to Closed-Set settings or to GAN-generated images, relying on fragile frequency-based "fingerprint" features. By contrast, we propose a simple yet effective framework that incorporates features from large pre-trained foundation models to perform Open-Set origin attribution of synthetic images produced by various generative models, including Diffusion Models. We show that our method leads to remarkable attribution performance, even in the low-data regime, exceeding the performance of existing methods and generalizes better on images obtained from a diverse set of architectures. We make the code publicly available at: https://github.com/ciodar/UniversalAttribution.

Related papers

• BOSC: A Backdoor-based Framework for Open Set Synthetic Image Attribution [22.81354665006496]
  Synthetic image attribution addresses the problem of tracing back the origin of images produced by generative models. We propose a framework for open set attribution of synthetic images, named BOSC, that relies on the concept of backdoor attacks.
  arXiv  Detail & Related papers  (2024-05-19T09:17:43Z)
• Diversified in-domain synthesis with efficient fine-tuning for few-shot classification [64.86872227580866]
  Few-shot image classification aims to learn an image classifier using only a small set of labeled examples per class. We propose DISEF, a novel approach which addresses the generalization challenge in few-shot learning using synthetic data. We validate our method in ten different benchmarks, consistently outperforming baselines and establishing a new state-of-the-art for few-shot classification.
  arXiv  Detail & Related papers  (2023-12-05T17:18:09Z)
• Diversify, Don't Fine-Tune: Scaling Up Visual Recognition Training with Synthetic Images [37.29348016920314]
  We present a new framework leveraging off-the-shelf generative models to generate synthetic training images. We address class name ambiguity, lack of diversity in naive prompts, and domain shifts. Our framework consistently enhances recognition model performance with more synthetic data.
  arXiv  Detail & Related papers  (2023-12-04T18:35:27Z)
• Steered Diffusion: A Generalized Framework for Plug-and-Play Conditional Image Synthesis [62.07413805483241]
  Steered Diffusion is a framework for zero-shot conditional image generation using a diffusion model trained for unconditional generation. We present experiments using steered diffusion on several tasks including inpainting, colorization, text-guided semantic editing, and image super-resolution.
  arXiv  Detail & Related papers  (2023-09-30T02:03:22Z)
• Improving Few-shot Image Generation by Structural Discrimination and Textural Modulation [10.389698647141296]
  Few-shot image generation aims to produce plausible and diverse images for one category given a few images from this category. Existing approaches either globally interpolate different images or fuse local representations with pre-defined coefficients. This paper proposes a novel mechanism to inject external semantic signals into internal local representations.
  arXiv  Detail & Related papers  (2023-08-30T16:10:21Z)
• Parents and Children: Distinguishing Multimodal DeepFakes from Natural Images [60.34381768479834]
  Recent advancements in diffusion models have enabled the generation of realistic deepfakes from textual prompts in natural language. We pioneer a systematic study on deepfake detection generated by state-of-the-art diffusion models.
  arXiv  Detail & Related papers  (2023-04-02T10:25:09Z)
• Progressive Open Space Expansion for Open-Set Model Attribution [19.985618498466042]
  We focus on a challenging task, namely Open-Set Model Attribution (OSMA), to simultaneously attribute images to known models and identify those from unknown ones. Compared to existing open-set recognition (OSR) tasks, OSMA is more challenging as the distinction between images from known and unknown models may only lie in visually imperceptible traces. We propose a Progressive Open Space Expansion (POSE) solution, which simulates open-set samples that maintain the same semantics as closed-set samples but embedded with different imperceptible traces.
  arXiv  Detail & Related papers  (2023-03-13T05:53:11Z)
• Person Image Synthesis via Denoising Diffusion Model [116.34633988927429]
  We show how denoising diffusion models can be applied for high-fidelity person image synthesis. Our results on two large-scale benchmarks and a user study demonstrate the photorealism of our proposed approach under challenging scenarios.
  arXiv  Detail & Related papers  (2022-11-22T18:59:50Z)
• DiVAE: Photorealistic Images Synthesis with Denoising Diffusion Decoder [73.1010640692609]
  We propose a VQ-VAE architecture model with a diffusion decoder (DiVAE) to work as the reconstructing component in image synthesis. Our model achieves state-of-the-art results and generates more photorealistic images specifically.
  arXiv  Detail & Related papers  (2022-06-01T10:39:12Z)
• InvGAN: Invertible GANs [88.58338626299837]
  InvGAN, short for Invertible GAN, successfully embeds real images to the latent space of a high quality generative model. This allows us to perform image inpainting, merging, and online data augmentation.
  arXiv  Detail & Related papers  (2021-12-08T21:39:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.