RealEra: Semantic-level Concept Erasure via Neighbor-Concept Mining

• URL: http://arxiv.org/abs/2410.09140v1
• Date: Fri, 11 Oct 2024 17:55:30 GMT
• Title: RealEra: Semantic-level Concept Erasure via Neighbor-Concept Mining
• Authors: Yufan Liu, Jinyang An, Wanqian Zhang, Ming Li, Dayan Wu, Jingzi Gu, Zheng Lin, Weiping Wang,
• Abstract summary: Concept erasure has been proposed to remove the model's knowledge about protected and inappropriate concepts. We propose RealEra to address this "concept residue" issue. We show that RealEra outperforms previous concept erasing methods in terms of superior erasing efficacy, specificity, and generality.
• Score: 25.769144703607214
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The remarkable development of text-to-image generation models has raised notable security concerns, such as the infringement of portrait rights and the generation of inappropriate content. Concept erasure has been proposed to remove the model's knowledge about protected and inappropriate concepts. Although many methods have tried to balance the efficacy (erasing target concepts) and specificity (retaining irrelevant concepts), they can still generate abundant erasure concepts under the steering of semantically related inputs. In this work, we propose RealEra to address this "concept residue" issue. Specifically, we first introduce the mechanism of neighbor-concept mining, digging out the associated concepts by adding random perturbation into the embedding of erasure concept, thus expanding the erasing range and eliminating the generations even through associated concept inputs. Furthermore, to mitigate the negative impact on the generation of irrelevant concepts caused by the expansion of erasure scope, RealEra preserves the specificity through the beyond-concept regularization. This makes irrelevant concepts maintain their corresponding spatial position, thereby preserving their normal generation performance. We also employ the closed-form solution to optimize weights of U-Net for the cross-attention alignment, as well as the prediction noise alignment with the LoRA module. Extensive experiments on multiple benchmarks demonstrate that RealEra outperforms previous concept erasing methods in terms of superior erasing efficacy, specificity, and generality. More details are available on our project page https://realerasing.github.io/RealEra/ .

Related papers

• Fantastic Targets for Concept Erasure in Diffusion Models and Where To Find Them [21.386640828092524]
  Concept erasure has emerged as a promising technique for mitigating the risk of harmful content generation in diffusion models. We propose the Adaptive Guided Erasure (AGE) method, which emphdynamically selects optimal target concepts tailored to each undesirable concept. Results show that AGE significantly outperforms state-of-the-art erasure methods on preserving unrelated concepts while maintaining effective erasure performance.
  arXiv  Detail & Related papers  (2025-01-31T08:17:23Z)
• ACE: Anti-Editing Concept Erasure in Text-to-Image Models [73.00930293474009]
  Existing concept erasure methods achieve superior results in preventing the production of erased concept from prompts. We propose an Anti-Editing Concept Erasure (ACE) method, which not only erases the target concept during generation but also filters out it during editing.
  arXiv  Detail & Related papers  (2025-01-03T04:57:27Z)
• DuMo: Dual Encoder Modulation Network for Precise Concept Erasure [75.05165577219425]
  We propose our Dual encoder Modulation network (DuMo) which achieves precise erasure of inappropriate target concepts with minimum impairment to non-target concepts. Our method achieves state-of-the-art performance on Explicit Content Erasure, Cartoon Concept Removal and Artistic Style Erasure, clearly outperforming alternative methods.
  arXiv  Detail & Related papers  (2025-01-02T07:47:34Z)
• Memories of Forgotten Concepts [16.53173953073833]
  We show that erased concept images can be generated using the right latent. We extend this to demonstrate that for every image from the erased concept set, we can generate many seeds that generate the erased concept. Given the vast space of latents capable of generating ablated concept images, our results suggest that fully erasing concept information may be intractable.
  arXiv  Detail & Related papers  (2024-12-01T12:12:24Z)
• How to Continually Adapt Text-to-Image Diffusion Models for Flexible Customization? [91.49559116493414]
  We propose a novel Concept-Incremental text-to-image Diffusion Model (CIDM) It can resolve catastrophic forgetting and concept neglect to learn new customization tasks in a concept-incremental manner. Experiments validate that our CIDM surpasses existing custom diffusion models.
  arXiv  Detail & Related papers  (2024-10-23T06:47:29Z)
• Reliable and Efficient Concept Erasure of Text-to-Image Diffusion Models [76.39651111467832]
  We introduce Reliable and Efficient Concept Erasure (RECE), a novel approach that modifies the model in 3 seconds without necessitating additional fine-tuning. To mitigate inappropriate content potentially represented by derived embeddings, RECE aligns them with harmless concepts in cross-attention layers. The derivation and erasure of new representation embeddings are conducted iteratively to achieve a thorough erasure of inappropriate concepts.
  arXiv  Detail & Related papers  (2024-07-17T08:04:28Z)
• MACE: Mass Concept Erasure in Diffusion Models [11.12833789743765]
  We introduce MACE, a finetuning framework for the task of mass concept erasure. This task aims to prevent models from generating images that embody unwanted concepts when prompted. We conduct extensive evaluations of MACE against prior methods across four different tasks.
  arXiv  Detail & Related papers  (2024-03-10T08:50:56Z)
• Separable Multi-Concept Erasure from Diffusion Models [52.51972530398691]
  We propose a Separable Multi-concept Eraser (SepME) to eliminate unsafe concepts from large-scale diffusion models. The latter separates optimizable model weights, making each weight increment correspond to a specific concept erasure. Extensive experiments indicate the efficacy of our approach in eliminating concepts, preserving model performance, and offering flexibility in the erasure or recovery of various concepts.
  arXiv  Detail & Related papers  (2024-02-03T11:10:57Z)
• Implicit Concept Removal of Diffusion Models [92.55152501707995]
  Text-to-image (T2I) diffusion models often inadvertently generate unwanted concepts such as watermarks and unsafe images. We present the Geom-Erasing, a novel concept removal method based on the geometric-driven control.
  arXiv  Detail & Related papers  (2023-10-09T17:13:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.