Concept Replacer: Replacing Sensitive Concepts in Diffusion Models via Precision Localization

• URL: http://arxiv.org/abs/2412.01244v2
• Date: Tue, 03 Dec 2024 04:25:48 GMT
• Title: Concept Replacer: Replacing Sensitive Concepts in Diffusion Models via Precision Localization
• Authors: Lingyun Zhang, Yu Xie, Yanwei Fu, Ping Chen,
• Abstract summary: Large-scale diffusion models produce high-quality images but often generate unwanted content, such as sexually explicit or violent content.<n>We propose a novel approach for targeted concept replacing in diffusion models, enabling specific concepts to be removed without affecting non-target areas.<n>Our method introduces a dedicated concept localizer for precisely identifying the target concept during the denoising process, trained with few-shot learning to require minimal labeled data.<n>Within the identified region, we introduce a training-free Dual Prompts Cross-Attention (DPCA) module to substitute the target concept, ensuring minimal disruption to surrounding content.
• Score: 48.20360860166279
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: As large-scale diffusion models continue to advance, they excel at producing high-quality images but often generate unwanted content, such as sexually explicit or violent content. Existing methods for concept removal generally guide the image generation process but can unintentionally modify unrelated regions, leading to inconsistencies with the original model. We propose a novel approach for targeted concept replacing in diffusion models, enabling specific concepts to be removed without affecting non-target areas. Our method introduces a dedicated concept localizer for precisely identifying the target concept during the denoising process, trained with few-shot learning to require minimal labeled data. Within the identified region, we introduce a training-free Dual Prompts Cross-Attention (DPCA) module to substitute the target concept, ensuring minimal disruption to surrounding content. We evaluate our method on concept localization precision and replacement efficiency. Experimental results demonstrate that our method achieves superior precision in localizing target concepts and performs coherent concept replacement with minimal impact on non-target areas, outperforming existing approaches.

Related papers

• ACE: Attentional Concept Erasure in Diffusion Models [0.0]
  Attentional Concept Erasure integrates a closed-form attention manipulation with lightweight fine-tuning. We show that ACE achieves state-of-the-art concept removal efficacy and robustness. Compared to prior methods, ACE better balances generality (erasing concept and related terms) and specificity (preserving unrelated content)
  arXiv  Detail & Related papers  (2025-04-16T08:16:28Z)
• CRCE: Coreference-Retention Concept Erasure in Text-to-Image Diffusion Models [19.074434401274285]
  We introduce CRCE, a novel concept erasure framework. By explicitly modeling coreferential and retained concepts semantically, CRCE enables more precise concept removal. Experiments demonstrate that CRCE outperforms existing methods on diverse erasure tasks.
  arXiv  Detail & Related papers  (2025-03-18T13:09:01Z)
• Continual Unlearning for Foundational Text-to-Image Models without Generalization Erosion [56.35484513848296]
  This research introduces continual unlearning', a novel paradigm that enables the targeted removal of multiple specific concepts from foundational generative models. We propose Decremental Unlearning without Generalization Erosion (DUGE) algorithm which selectively unlearns the generation of undesired concepts.
  arXiv  Detail & Related papers  (2025-03-17T23:17:16Z)
• Localized Concept Erasure for Text-to-Image Diffusion Models Using Training-Free Gated Low-Rank Adaptation [12.098673137752861]
  Fine-tuning based concept erasing has demonstrated promising results in preventing generation of harmful contents from text-to-image diffusion models. We introduce a framework called localized concept erasure, which allows for the deletion of only the specific area containing the target concept in the image. We propose a training-free approach, dubbed Gated Low-rank adaptation for Concept Erasure (GLoCE), that injects a lightweight module into the diffusion model.
  arXiv  Detail & Related papers  (2025-03-16T04:53:20Z)
• Modular Customization of Diffusion Models via Blockwise-Parameterized Low-Rank Adaptation [73.16975077770765]
  Modular customization is essential for applications like concept stylization and multi-concept customization. Instant merging methods often cause identity loss and interference of individual merged concepts. We propose BlockLoRA, an instant merging method designed to efficiently combine multiple concepts while accurately preserving individual concepts' identity.
  arXiv  Detail & Related papers  (2025-03-11T16:10:36Z)
• Few-Shot Image Generation by Conditional Relaxing Diffusion Inversion [37.18537753482751]
  Conditional Diffusion Relaxing Inversion (CRDI) is designed to enhance distribution diversity in synthetic image generation. CRDI does not rely on fine-tuning based on only a few samples. It focuses on reconstructing each target image instance and expanding diversity through few-shot learning.
  arXiv  Detail & Related papers  (2024-07-09T21:58:26Z)
• Six-CD: Benchmarking Concept Removals for Benign Text-to-image Diffusion Models [58.74606272936636]
  Text-to-image (T2I) diffusion models have shown exceptional capabilities in generating images that closely correspond to textual prompts. The models could be exploited for malicious purposes, such as generating images with violence or nudity, or creating unauthorized portraits of public figures in inappropriate contexts. concept removal methods have been proposed to modify diffusion models to prevent the generation of malicious and unwanted concepts.
  arXiv  Detail & Related papers  (2024-06-21T03:58:44Z)
• Unlearning Concepts in Diffusion Model via Concept Domain Correction and Concept Preserving Gradient [20.091446060893638]
  This paper proposes a concept domain correction framework for unlearning concepts in diffusion models. By aligning the output domains of sensitive concepts and anchor concepts through adversarial training, we enhance the generalizability of the unlearning results.
  arXiv  Detail & Related papers  (2024-05-24T07:47:36Z)
• Personalized Residuals for Concept-Driven Text-to-Image Generation [29.052642845759372]
  We present personalized residuals and localized attention-guided sampling for efficient concept-driven generation using text-to-image diffusion models. We show that personalized residuals effectively capture the identity of a concept in 3 minutes on a single GPU without the use of regularization images.
  arXiv  Detail & Related papers  (2024-05-21T17:59:01Z)
• Compositional Inversion for Stable Diffusion Models [64.79261401944994]
  Inversion methods generate personalized images by incorporating concepts of interest provided by user images. Existing methods often suffer from overfitting issues, where the dominant presence of inverted concepts leads to the absence of other desired concepts. We propose a method that guides the inversion process towards the core distribution for compositional embeddings.
  arXiv  Detail & Related papers  (2023-12-13T10:57:46Z)
• Implicit Concept Removal of Diffusion Models [92.55152501707995]
  Text-to-image (T2I) diffusion models often inadvertently generate unwanted concepts such as watermarks and unsafe images. We present the Geom-Erasing, a novel concept removal method based on the geometric-driven control.
  arXiv  Detail & Related papers  (2023-10-09T17:13:10Z)
• Phasic Content Fusing Diffusion Model with Directional Distribution Consistency for Few-Shot Model Adaption [73.98706049140098]
  We propose a novel phasic content fusing few-shot diffusion model with directional distribution consistency loss. Specifically, we design a phasic training strategy with phasic content fusion to help our model learn content and style information when t is large. Finally, we propose a cross-domain structure guidance strategy that enhances structure consistency during domain adaptation.
  arXiv  Detail & Related papers  (2023-09-07T14:14:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.