Contrasting Deepfakes Diffusion via Contrastive Learning and Global-Local Similarities

• URL: http://arxiv.org/abs/2407.20337v1
• Date: Mon, 29 Jul 2024 18:00:10 GMT
• Title: Contrasting Deepfakes Diffusion via Contrastive Learning and Global-Local Similarities
• Authors: Lorenzo Baraldi, Federico Cocchi, Marcella Cornia, Lorenzo Baraldi, Alessandro Nicolosi, Rita Cucchiara,
• Abstract summary: Contrastive Deepfake Embeddings (CoDE) is a novel embedding space specifically designed for deepfake detection. CoDE is trained via contrastive learning by additionally enforcing global-local similarities.
• Score: 88.398085358514
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Discerning between authentic content and that generated by advanced AI methods has become increasingly challenging. While previous research primarily addresses the detection of fake faces, the identification of generated natural images has only recently surfaced. This prompted the recent exploration of solutions that employ foundation vision-and-language models, like CLIP. However, the CLIP embedding space is optimized for global image-to-text alignment and is not inherently designed for deepfake detection, neglecting the potential benefits of tailored training and local image features. In this study, we propose CoDE (Contrastive Deepfake Embeddings), a novel embedding space specifically designed for deepfake detection. CoDE is trained via contrastive learning by additionally enforcing global-local similarities. To sustain the training of our model, we generate a comprehensive dataset that focuses on images generated by diffusion models and encompasses a collection of 9.2 million images produced by using four different generators. Experimental results demonstrate that CoDE achieves state-of-the-art accuracy on the newly collected dataset, while also showing excellent generalization capabilities to unseen image generators. Our source code, trained models, and collected dataset are publicly available at: https://github.com/aimagelab/CoDE.

Related papers

• MFCLIP: Multi-modal Fine-grained CLIP for Generalizable Diffusion Face Forgery Detection [64.29452783056253]
  The rapid development of photo-realistic face generation methods has raised significant concerns in society and academia. Although existing approaches mainly capture face forgery patterns using image modality, other modalities like fine-grained noises and texts are not fully explored. We propose a novel multi-modal fine-grained CLIP (MFCLIP) model, which mines comprehensive and fine-grained forgery traces across image-noise modalities.
  arXiv  Detail & Related papers  (2024-09-15T13:08:59Z)
• GenFace: A Large-Scale Fine-Grained Face Forgery Benchmark and Cross Appearance-Edge Learning [50.7702397913573]
  The rapid advancement of photorealistic generators has reached a critical juncture where the discrepancy between authentic and manipulated images is increasingly indistinguishable. Although there have been a number of publicly available face forgery datasets, the forgery faces are mostly generated using GAN-based synthesis technology. We propose a large-scale, diverse, and fine-grained high-fidelity dataset, namely GenFace, to facilitate the advancement of deepfake detection.
  arXiv  Detail & Related papers  (2024-02-03T03:13:50Z)
• Rethinking the Up-Sampling Operations in CNN-based Generative Network for Generalizable Deepfake Detection [86.97062579515833]
  We introduce the concept of Neighboring Pixel Relationships(NPR) as a means to capture and characterize the generalized structural artifacts stemming from up-sampling operations. A comprehensive analysis is conducted on an open-world dataset, comprising samples generated by tft28 distinct generative models. This analysis culminates in the establishment of a novel state-of-the-art performance, showcasing a remarkable tft11.6% improvement over existing methods.
  arXiv  Detail & Related papers  (2023-12-16T14:27:06Z)
• DiAD: A Diffusion-based Framework for Multi-class Anomaly Detection [55.48770333927732]
  We propose a Difusion-based Anomaly Detection (DiAD) framework for multi-class anomaly detection. It consists of a pixel-space autoencoder, a latent-space Semantic-Guided (SG) network with a connection to the stable diffusion's denoising network, and a feature-space pre-trained feature extractor. Experiments on MVTec-AD and VisA datasets demonstrate the effectiveness of our approach.
  arXiv  Detail & Related papers  (2023-12-11T18:38:28Z)
• Parents and Children: Distinguishing Multimodal DeepFakes from Natural Images [60.34381768479834]
  Recent advancements in diffusion models have enabled the generation of realistic deepfakes from textual prompts in natural language. We pioneer a systematic study on deepfake detection generated by state-of-the-art diffusion models.
  arXiv  Detail & Related papers  (2023-04-02T10:25:09Z)
• GLFF: Global and Local Feature Fusion for AI-synthesized Image Detection [29.118321046339656]
  We propose a framework to learn rich and discriminative representations by combining multi-scale global features from the whole image with refined local features from informative patches for AI synthesized image detection. GLFF fuses information from two branches: the global branch to extract multi-scale semantic features and the local branch to select informative patches for detailed local artifacts extraction.
  arXiv  Detail & Related papers  (2022-11-16T02:03:20Z)
• Fusing Global and Local Features for Generalized AI-Synthesized Image Detection [31.35052580048599]
  We design a two-branch model to combine global spatial information from the whole image and local informative features from patches selected by a novel patch selection module. We collect a highly diverse dataset synthesized by 19 models with various objects and resolutions to evaluate our model.
  arXiv  Detail & Related papers  (2022-03-26T01:55:37Z)
• Towards Discovery and Attribution of Open-world GAN Generated Images [18.10496076534083]
  We present an iterative algorithm for discovering images generated from previously unseen GANs. Our algorithm consists of multiple components including network training, out-of-distribution detection, clustering, merge and refine steps. Our experiments demonstrate the effectiveness of our approach to discover new GANs and can be used in an open-world setup.
  arXiv  Detail & Related papers  (2021-05-10T18:00:13Z)
• DeepFake Detection by Analyzing Convolutional Traces [0.0]
  We focus on the analysis of Deepfakes of human faces with the objective of creating a new detection method. The proposed technique, by means of an Expectation Maximization (EM) algorithm, extracts a set of local features specifically addressed to model the underlying convolutional generative process. Results demonstrated the effectiveness of the technique in distinguishing the different architectures and the corresponding generation process.
  arXiv  Detail & Related papers  (2020-04-22T09:02:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.