MemBench: Memorized Image Trigger Prompt Dataset for Diffusion Models

• URL: http://arxiv.org/abs/2407.17095v2
• Date: Mon, 30 Sep 2024 17:02:48 GMT
• Title: MemBench: Memorized Image Trigger Prompt Dataset for Diffusion Models
• Authors: Chunsan Hong, Tae-Hyun Oh, Minhyuk Sung,
• Abstract summary: We present MemBench, the first benchmark for evaluating image memorization mitigation methods. We present metrics evaluating on both trigger prompts and general prompts, so that we can see whether mitigation methods address the memorization issue.
• Score: 28.64962324938148
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Diffusion models have achieved remarkable success in Text-to-Image generation tasks, leading to the development of many commercial models. However, recent studies have reported that diffusion models often generate replicated images in train data when triggered by specific prompts, potentially raising social issues ranging from copyright to privacy concerns. To sidestep the memorization, there have been recent studies for developing memorization mitigation methods for diffusion models. Nevertheless, the lack of benchmarks impedes the assessment of the true effectiveness of these methods. In this work, we present MemBench, the first benchmark for evaluating image memorization mitigation methods. Our benchmark includes a large number of memorized image trigger prompts in various Text-to-Image diffusion models. Furthermore, in contrast to the prior work evaluating mitigation performance only on trigger prompts, we present metrics evaluating on both trigger prompts and general prompts, so that we can see whether mitigation methods address the memorization issue while maintaining performance for general prompts. This is an important development considering the practical applications which previous works have overlooked. Through evaluation on MemBench, we verify that the performance of existing image memorization mitigation methods is still insufficient for application to diffusion models. The code and datasets are available at https://github.com/chunsanHong/MemBench\_code.

Related papers

• Learning Diffusion Model from Noisy Measurement using Principled Expectation-Maximization Method [9.173055778539641]
  We propose a principled expectation-maximization (EM) framework that iteratively learns diffusion models from noisy data with arbitrary corruption types. Our framework employs a plug-and-play Monte Carlo method to accurately estimate clean images from noisy measurements, followed by training the diffusion model using the reconstructed images.
  arXiv  Detail & Related papers  (2024-10-15T03:54:59Z)
• Detecting, Explaining, and Mitigating Memorization in Diffusion Models [49.438362005962375]
  We introduce a straightforward yet effective method for detecting memorized prompts by inspecting the magnitude of text-conditional predictions. Our proposed method seamlessly integrates without disrupting sampling algorithms, and delivers high accuracy even at the first generation step. Building on our detection strategy, we unveil an explainable approach that shows the contribution of individual words or tokens to memorization.
  arXiv  Detail & Related papers  (2024-07-31T16:13:29Z)
• Memorized Images in Diffusion Models share a Subspace that can be Located and Deleted [15.162296378581853]
  Large-scale text-to-image diffusion models excel in generating high-quality images from textual inputs. Concerns arise as research indicates their tendency to memorize and replicate training data. Efforts within the text-to-image community to address memorization explore causes such as data duplication, replicated captions, or trigger tokens.
  arXiv  Detail & Related papers  (2024-06-01T15:47:13Z)
• Could It Be Generated? Towards Practical Analysis of Memorization in Text-To-Image Diffusion Models [39.607005089747936]
  We perform practical analysis of memorization in text-to-image diffusion models. We identify three necessary conditions of memorization, respectively similarity, existence and probability. We then reveal the correlation between the model's prediction error and image replication.
  arXiv  Detail & Related papers  (2024-05-09T15:32:00Z)
• Unveiling and Mitigating Memorization in Text-to-image Diffusion Models through Cross Attention [62.671435607043875]
  Research indicates that text-to-image diffusion models replicate images from their training data, raising tremendous concerns about potential copyright infringement and privacy risks. We reveal that during memorization, the cross-attention tends to focus disproportionately on the embeddings of specific tokens. We introduce an innovative approach to detect and mitigate memorization in diffusion models.
  arXiv  Detail & Related papers  (2024-03-17T01:27:00Z)
• What do larger image classifiers memorise? [64.01325988398838]
  We show that training examples exhibit an unexpectedly diverse set of memorisation trajectories across model sizes. We find that knowledge distillation, an effective and popular model compression technique, tends to inhibit memorisation, while also improving generalisation.
  arXiv  Detail & Related papers  (2023-10-09T01:52:07Z)
• DIAGNOSIS: Detecting Unauthorized Data Usages in Text-to-image Diffusion Models [79.71665540122498]
  We propose a method for detecting unauthorized data usage by planting the injected content into the protected dataset. Specifically, we modify the protected images by adding unique contents on these images using stealthy image warping functions. By analyzing whether the model has memorized the injected content, we can detect models that had illegally utilized the unauthorized data.
  arXiv  Detail & Related papers  (2023-07-06T16:27:39Z)
• Understanding and Mitigating Copying in Diffusion Models [53.03978584040557]
  Images generated by diffusion models like Stable Diffusion are increasingly widespread. Recent works and even lawsuits have shown that these models are prone to replicating their training data, unbeknownst to the user.
  arXiv  Detail & Related papers  (2023-05-31T17:58:02Z)
• Always Be Dreaming: A New Approach for Data-Free Class-Incremental Learning [73.24988226158497]
  We consider the high-impact problem of Data-Free Class-Incremental Learning (DFCIL) We propose a novel incremental distillation strategy for DFCIL, contributing a modified cross-entropy training and importance-weighted feature distillation. Our method results in up to a 25.1% increase in final task accuracy (absolute difference) compared to SOTA DFCIL methods for common class-incremental benchmarks.
  arXiv  Detail & Related papers  (2021-06-17T17:56:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.