Measuring the Success of Diffusion Models at Imitating Human Artists
- URL: http://arxiv.org/abs/2307.04028v1
- Date: Sat, 8 Jul 2023 18:31:25 GMT
- Title: Measuring the Success of Diffusion Models at Imitating Human Artists
- Authors: Stephen Casper, Zifan Guo, Shreya Mogulothu, Zachary Marinov, Chinmay
Deshpande, Rui-Jie Yew, Zheng Dai, Dylan Hadfield-Menell
- Abstract summary: We show how to measure a model's ability to imitate specific artists.
We use Contrastive Language-Image Pretrained (CLIP) encoders to classify images in a zero-shot fashion.
We also show that a sample of the artist's work can be matched to these imitation images with a high degree of statistical reliability.
- Score: 7.007492782620398
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Modern diffusion models have set the state-of-the-art in AI image generation.
Their success is due, in part, to training on Internet-scale data which often
includes copyrighted work. This prompts questions about the extent to which
these models learn from, imitate, or copy the work of human artists. This work
suggests that tying copyright liability to the capabilities of the model may be
useful given the evolving ecosystem of generative models. Specifically, much of
the legal analysis of copyright and generative systems focuses on the use of
protected data for training. As a result, the connections between data,
training, and the system are often obscured. In our approach, we consider
simple image classification techniques to measure a model's ability to imitate
specific artists. Specifically, we use Contrastive Language-Image Pretrained
(CLIP) encoders to classify images in a zero-shot fashion. Our process first
prompts a model to imitate a specific artist. Then, we test whether CLIP can be
used to reclassify the artist (or the artist's work) from the imitation. If
these tests match the imitation back to the original artist, this suggests the
model can imitate that artist's expression. Our approach is simple and
quantitative. Furthermore, it uses standard techniques and does not require
additional training. We demonstrate our approach with an audit of Stable
Diffusion's capacity to imitate 70 professional digital artists with
copyrighted work online. When Stable Diffusion is prompted to imitate an artist
from this set, we find that the artist can be identified from the imitation
with an average accuracy of 81.0%. Finally, we also show that a sample of the
artist's work can be matched to these imitation images with a high degree of
statistical reliability. Overall, these results suggest that Stable Diffusion
is broadly successful at imitating individual human artists.
Related papers
- How Many Van Goghs Does It Take to Van Gogh? Finding the Imitation Threshold [50.33428591760124]
We study the relationship between a concept's frequency in the training dataset and the ability of a model to imitate it.
We propose an efficient approach that estimates the imitation threshold without incurring the colossal cost of training multiple models from scratch.
arXiv Detail & Related papers (2024-10-19T06:28:14Z) - Not Every Image is Worth a Thousand Words: Quantifying Originality in Stable Diffusion [21.252145402613472]
This work addresses the challenge of quantifying originality in text-to-image (T2I) generative diffusion models.
We propose a method that leverages textual inversion to measure the originality of an image based on the number of tokens required for its reconstruction by the model.
arXiv Detail & Related papers (2024-08-15T14:42:02Z) - Rethinking Artistic Copyright Infringements in the Era of Text-to-Image Generative Models [47.19481598385283]
ArtSavant is a tool to determine the unique style of an artist by comparing it to a reference dataset of works from WikiArt.
We then perform a large-scale empirical study to provide quantitative insight on the prevalence of artistic style copying across 3 popular text-to-image generative models.
arXiv Detail & Related papers (2024-04-11T17:59:43Z) - A Dataset and Benchmark for Copyright Infringement Unlearning from Text-to-Image Diffusion Models [52.49582606341111]
Copyright law confers creators the exclusive rights to reproduce, distribute, and monetize their creative works.
Recent progress in text-to-image generation has introduced formidable challenges to copyright enforcement.
We introduce a novel pipeline that harmonizes CLIP, ChatGPT, and diffusion models to curate a dataset.
arXiv Detail & Related papers (2024-01-04T11:14:01Z) - Not with my name! Inferring artists' names of input strings employed by
Diffusion Models [8.692128987695423]
Diffusion Models (DM) are highly effective at generating realistic, high-quality images.
However, these models lack creativity and merely compose outputs based on their training data.
In this paper, a preliminary study to infer the probability of use of an artist's name in the input string of a generated image is presented.
arXiv Detail & Related papers (2023-07-25T14:18:58Z) - DIAGNOSIS: Detecting Unauthorized Data Usages in Text-to-image Diffusion Models [79.71665540122498]
We propose a method for detecting unauthorized data usage by planting the injected content into the protected dataset.
Specifically, we modify the protected images by adding unique contents on these images using stealthy image warping functions.
By analyzing whether the model has memorized the injected content, we can detect models that had illegally utilized the unauthorized data.
arXiv Detail & Related papers (2023-07-06T16:27:39Z) - Learning to Evaluate the Artness of AI-generated Images [64.48229009396186]
ArtScore is a metric designed to evaluate the degree to which an image resembles authentic artworks by artists.
We employ pre-trained models for photo and artwork generation, resulting in a series of mixed models.
This dataset is then employed to train a neural network that learns to estimate quantized artness levels of arbitrary images.
arXiv Detail & Related papers (2023-05-08T17:58:27Z) - Diffusion Art or Digital Forgery? Investigating Data Replication in
Diffusion Models [53.03978584040557]
We study image retrieval frameworks that enable us to compare generated images with training samples and detect when content has been replicated.
Applying our frameworks to diffusion models trained on multiple datasets including Oxford flowers, Celeb-A, ImageNet, and LAION, we discuss how factors such as training set size impact rates of content replication.
arXiv Detail & Related papers (2022-12-07T18:58:02Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.