Receler: Reliable Concept Erasing of Text-to-Image Diffusion Models via Lightweight Erasers

• URL: http://arxiv.org/abs/2311.17717v3
• Date: Thu, 18 Jul 2024 07:23:03 GMT
• Title: Receler: Reliable Concept Erasing of Text-to-Image Diffusion Models via Lightweight Erasers
• Authors: Chi-Pin Huang, Kai-Po Chang, Chung-Ting Tsai, Yung-Hsuan Lai, Fu-En Yang, Yu-Chiang Frank Wang,
• Abstract summary: Concept erasure in text-to-image diffusion models aims to disable pre-trained diffusion models from generating images related to a target concept. We propose Reliable Concept Erasing via Lightweight Erasers (Receler)
• Score: 24.64639078273091
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Concept erasure in text-to-image diffusion models aims to disable pre-trained diffusion models from generating images related to a target concept. To perform reliable concept erasure, the properties of robustness and locality are desirable. The former refrains the model from producing images associated with the target concept for any paraphrased or learned prompts, while the latter preserves its ability in generating images with non-target concepts. In this paper, we propose Reliable Concept Erasing via Lightweight Erasers (Receler). It learns a lightweight Eraser to perform concept erasing while satisfying the above desirable properties through the proposed concept-localized regularization and adversarial prompt learning scheme. Experiments with various concepts verify the superiority of Receler over previous methods.

Related papers

• Towards Lifelong Few-Shot Customization of Text-to-Image Diffusion [50.26583654615212]
  Lifelong few-shot customization for text-to-image diffusion aims to continually generalize existing models for new tasks with minimal data. In this study, we identify and categorize the catastrophic forgetting problems into two folds: relevant concepts forgetting and previous concepts forgetting. Unlike existing methods that rely on additional real data or offline replay of original concept data, our approach enables on-the-fly knowledge distillation to retain the previous concepts while learning new ones.
  arXiv  Detail & Related papers  (2024-11-08T12:58:48Z)
• Reliable and Efficient Concept Erasure of Text-to-Image Diffusion Models [76.39651111467832]
  We introduce Reliable and Efficient Concept Erasure (RECE), a novel approach that modifies the model in 3 seconds without necessitating additional fine-tuning. To mitigate inappropriate content potentially represented by derived embeddings, RECE aligns them with harmless concepts in cross-attention layers. The derivation and erasure of new representation embeddings are conducted iteratively to achieve a thorough erasure of inappropriate concepts.
  arXiv  Detail & Related papers  (2024-07-17T08:04:28Z)
• Six-CD: Benchmarking Concept Removals for Benign Text-to-image Diffusion Models [58.74606272936636]
  Text-to-image (T2I) diffusion models have shown exceptional capabilities in generating images that closely correspond to textual prompts. The models could be exploited for malicious purposes, such as generating images with violence or nudity, or creating unauthorized portraits of public figures in inappropriate contexts. concept removal methods have been proposed to modify diffusion models to prevent the generation of malicious and unwanted concepts.
  arXiv  Detail & Related papers  (2024-06-21T03:58:44Z)
• ConceptPrune: Concept Editing in Diffusion Models via Skilled Neuron Pruning [10.201633236997104]
  Large-scale text-to-image diffusion models have demonstrated impressive image-generation capabilities. We present ConceptPrune, wherein we first identify critical regions within pre-trained models responsible for generating undesirable concepts. Experiments across a range of concepts including artistic styles, nudity, object erasure, and gender debiasing demonstrate that target concepts can be efficiently erased by pruning a tiny fraction.
  arXiv  Detail & Related papers  (2024-05-29T16:19:37Z)
• Unlearning Concepts in Diffusion Model via Concept Domain Correction and Concept Preserving Gradient [20.091446060893638]
  This paper proposes a concept domain correction framework for unlearning concepts in diffusion models. By aligning the output domains of sensitive concepts and anchor concepts through adversarial training, we enhance the generalizability of the unlearning results.
  arXiv  Detail & Related papers  (2024-05-24T07:47:36Z)
• Erasing Concepts from Text-to-Image Diffusion Models with Few-shot Unlearning [0.0]
  We propose a novel concept-erasure method that updates the text encoder using few-shot unlearning. Our method can erase a concept within 10 s, making concept erasure more accessible than ever before.
  arXiv  Detail & Related papers  (2024-05-12T14:01:05Z)
• Separable Multi-Concept Erasure from Diffusion Models [52.51972530398691]
  We propose a Separable Multi-concept Eraser (SepME) to eliminate unsafe concepts from large-scale diffusion models. The latter separates optimizable model weights, making each weight increment correspond to a specific concept erasure. Extensive experiments indicate the efficacy of our approach in eliminating concepts, preserving model performance, and offering flexibility in the erasure or recovery of various concepts.
  arXiv  Detail & Related papers  (2024-02-03T11:10:57Z)
• Implicit Concept Removal of Diffusion Models [92.55152501707995]
  Text-to-image (T2I) diffusion models often inadvertently generate unwanted concepts such as watermarks and unsafe images. We present the Geom-Erasing, a novel concept removal method based on the geometric-driven control.
  arXiv  Detail & Related papers  (2023-10-09T17:13:10Z)
• Ablating Concepts in Text-to-Image Diffusion Models [57.9371041022838]
  Large-scale text-to-image diffusion models can generate high-fidelity images with powerful compositional ability. These models are typically trained on an enormous amount of Internet data, often containing copyrighted material, licensed images, and personal photos. We propose an efficient method of ablating concepts in the pretrained model, preventing the generation of a target concept.
  arXiv  Detail & Related papers  (2023-03-23T17:59:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.