Diffusion Art or Digital Forgery? Investigating Data Replication in Diffusion Models

• URL: http://arxiv.org/abs/2212.03860v3
• Date: Mon, 12 Dec 2022 18:52:13 GMT
• Title: Diffusion Art or Digital Forgery? Investigating Data Replication in Diffusion Models
• Authors: Gowthami Somepalli, Vasu Singla, Micah Goldblum, Jonas Geiping, Tom Goldstein
• Abstract summary: We study image retrieval frameworks that enable us to compare generated images with training samples and detect when content has been replicated. Applying our frameworks to diffusion models trained on multiple datasets including Oxford flowers, Celeb-A, ImageNet, and LAION, we discuss how factors such as training set size impact rates of content replication.
• Score: 53.03978584040557
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Cutting-edge diffusion models produce images with high quality and customizability, enabling them to be used for commercial art and graphic design purposes. But do diffusion models create unique works of art, or are they replicating content directly from their training sets? In this work, we study image retrieval frameworks that enable us to compare generated images with training samples and detect when content has been replicated. Applying our frameworks to diffusion models trained on multiple datasets including Oxford flowers, Celeb-A, ImageNet, and LAION, we discuss how factors such as training set size impact rates of content replication. We also identify cases where diffusion models, including the popular Stable Diffusion model, blatantly copy from their training data.

Related papers

• Efficient Shapley Values for Attributing Global Properties of Diffusion Models to Data Group [13.761241561734547]
  We develop a method to efficiently estimate Shapley values by leveraging model pruning and fine-tuning. We empirically demonstrate the utility of our method with three use cases: (i) global image quality for a DDPM trained on a CIFAR dataset, (ii) demographic diversity for an LDM trained on CelebA-HQ, and (iii) overall aesthetic quality for a Stable Diffusion model LoRA-finetuned on Post-Impressionist artworks.
  arXiv  Detail & Related papers  (2024-06-09T17:42:09Z)
• Diffusion Cocktail: Mixing Domain-Specific Diffusion Models for Diversified Image Generations [7.604214200457584]
  Diffusion Cocktail (Ditail) is a training-free method that transfers style and content information between multiple diffusion models. Ditail offers fine-grained control of the generation process, which enables flexible manipulations of styles and contents.
  arXiv  Detail & Related papers  (2023-12-12T00:53:56Z)
• The Journey, Not the Destination: How Data Guides Diffusion Models [75.19694584942623]
  Diffusion models trained on large datasets can synthesize photo-realistic images of remarkable quality and diversity. We propose a framework that: (i) provides a formal notion of data attribution in the context of diffusion models, and (ii) allows us to counterfactually validate such attributions.
  arXiv  Detail & Related papers  (2023-12-11T08:39:43Z)
• Stable Diffusion for Data Augmentation in COCO and Weed Datasets [5.81198182644659]
  This study utilized seven common categories and three widespread weed species to evaluate the efficiency of a stable diffusion model. Three techniques (i.e., Image-to-image translation, Dreambooth, and ControlNet) based on stable diffusion were leveraged for image generation with different focuses. Then, classification and detection tasks were conducted based on these synthetic images, whose performance was compared to the models trained on original images.
  arXiv  Detail & Related papers  (2023-12-07T02:23:32Z)
• Understanding and Mitigating Copying in Diffusion Models [53.03978584040557]
  Images generated by diffusion models like Stable Diffusion are increasingly widespread. Recent works and even lawsuits have shown that these models are prone to replicating their training data, unbeknownst to the user.
  arXiv  Detail & Related papers  (2023-05-31T17:58:02Z)
• Discffusion: Discriminative Diffusion Models as Few-shot Vision and Language Learners [88.07317175639226]
  We propose a novel approach, Discriminative Stable Diffusion (DSD), which turns pre-trained text-to-image diffusion models into few-shot discriminative learners. Our approach mainly uses the cross-attention score of a Stable Diffusion model to capture the mutual influence between visual and textual information.
  arXiv  Detail & Related papers  (2023-05-18T05:41:36Z)
• Diffusion Models as Masked Autoencoders [52.442717717898056]
  We revisit generatively pre-training visual representations in light of recent interest in denoising diffusion models. While directly pre-training with diffusion models does not produce strong representations, we condition diffusion models on masked input and formulate diffusion models as masked autoencoders (DiffMAE) We perform a comprehensive study on the pros and cons of design choices and build connections between diffusion models and masked autoencoders.
  arXiv  Detail & Related papers  (2023-04-06T17:59:56Z)
• Extracting Training Data from Diffusion Models [77.11719063152027]
  We show that diffusion models memorize individual images from their training data and emit them at generation time. With a generate-and-filter pipeline, we extract over a thousand training examples from state-of-the-art models. We train hundreds of diffusion models in various settings to analyze how different modeling and data decisions affect privacy.
  arXiv  Detail & Related papers  (2023-01-30T18:53:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.