Concept Replacer: Replacing Sensitive Concepts in Diffusion Models via Precision Localization

• URL: http://arxiv.org/abs/2412.01244v2
• Date: Tue, 03 Dec 2024 04:25:48 GMT
• Title: Concept Replacer: Replacing Sensitive Concepts in Diffusion Models via Precision Localization
• Authors: Lingyun Zhang, Yu Xie, Yanwei Fu, Ping Chen,
• Abstract summary: Large-scale diffusion models produce high-quality images but often generate unwanted content, such as sexually explicit or violent content. We propose a novel approach for targeted concept replacing in diffusion models, enabling specific concepts to be removed without affecting non-target areas. Our method introduces a dedicated concept localizer for precisely identifying the target concept during the denoising process, trained with few-shot learning to require minimal labeled data. Within the identified region, we introduce a training-free Dual Prompts Cross-Attention (DPCA) module to substitute the target concept, ensuring minimal disruption to surrounding content.
• Score: 48.20360860166279
• License:
• Abstract: As large-scale diffusion models continue to advance, they excel at producing high-quality images but often generate unwanted content, such as sexually explicit or violent content. Existing methods for concept removal generally guide the image generation process but can unintentionally modify unrelated regions, leading to inconsistencies with the original model. We propose a novel approach for targeted concept replacing in diffusion models, enabling specific concepts to be removed without affecting non-target areas. Our method introduces a dedicated concept localizer for precisely identifying the target concept during the denoising process, trained with few-shot learning to require minimal labeled data. Within the identified region, we introduce a training-free Dual Prompts Cross-Attention (DPCA) module to substitute the target concept, ensuring minimal disruption to surrounding content. We evaluate our method on concept localization precision and replacement efficiency. Experimental results demonstrate that our method achieves superior precision in localizing target concepts and performs coherent concept replacement with minimal impact on non-target areas, outperforming existing approaches.

Related papers

• TIDE: Training Locally Interpretable Domain Generalization Models Enables Test-time Correction [14.396966854171273]
  We consider the problem of single-source domain generalization. Existing methods typically rely on extensive augmentations to synthetically cover diverse domains during training. We propose an approach that compels models to leverage such local concepts during prediction.
  arXiv  Detail & Related papers  (2024-11-25T08:46:37Z)
• Reliable and Efficient Concept Erasure of Text-to-Image Diffusion Models [76.39651111467832]
  We introduce Reliable and Efficient Concept Erasure (RECE), a novel approach that modifies the model in 3 seconds without necessitating additional fine-tuning. To mitigate inappropriate content potentially represented by derived embeddings, RECE aligns them with harmless concepts in cross-attention layers. The derivation and erasure of new representation embeddings are conducted iteratively to achieve a thorough erasure of inappropriate concepts.
  arXiv  Detail & Related papers  (2024-07-17T08:04:28Z)
• Few-Shot Image Generation by Conditional Relaxing Diffusion Inversion [37.18537753482751]
  Conditional Diffusion Relaxing Inversion (CRDI) is designed to enhance distribution diversity in synthetic image generation. CRDI does not rely on fine-tuning based on only a few samples. It focuses on reconstructing each target image instance and expanding diversity through few-shot learning.
  arXiv  Detail & Related papers  (2024-07-09T21:58:26Z)
• Six-CD: Benchmarking Concept Removals for Benign Text-to-image Diffusion Models [58.74606272936636]
  Text-to-image (T2I) diffusion models have shown exceptional capabilities in generating images that closely correspond to textual prompts. The models could be exploited for malicious purposes, such as generating images with violence or nudity, or creating unauthorized portraits of public figures in inappropriate contexts. concept removal methods have been proposed to modify diffusion models to prevent the generation of malicious and unwanted concepts.
  arXiv  Detail & Related papers  (2024-06-21T03:58:44Z)
• Unlearning Concepts in Diffusion Model via Concept Domain Correction and Concept Preserving Gradient [20.698305103879232]
  We propose a novel concept domain correction framework named textbfDoCo (textbfDomaintextbfCorrection) By aligning the output domains of sensitive and anchor concepts through adversarial training, our approach ensures comprehensive unlearning of target concepts. We also introduce a concept-preserving gradient surgery technique that mitigates conflicting gradient components, thereby preserving the model's utility while unlearning specific concepts.
  arXiv  Detail & Related papers  (2024-05-24T07:47:36Z)
• Personalized Residuals for Concept-Driven Text-to-Image Generation [29.052642845759372]
  We present personalized residuals and localized attention-guided sampling for efficient concept-driven generation using text-to-image diffusion models. We show that personalized residuals effectively capture the identity of a concept in 3 minutes on a single GPU without the use of regularization images.
  arXiv  Detail & Related papers  (2024-05-21T17:59:01Z)
• Infusion: Preventing Customized Text-to-Image Diffusion from Overfitting [51.606819347636076]
  We analyze concept-agnostic overfitting, which undermines non-customized concept knowledge, and concept-specific overfitting, which is confined to customize on limited modalities. We propose Infusion, a T2I customization method that enables the learning of target concepts to avoid being constrained by limited training modalities.
  arXiv  Detail & Related papers  (2024-04-22T09:16:25Z)
• Compositional Inversion for Stable Diffusion Models [64.79261401944994]
  Inversion methods generate personalized images by incorporating concepts of interest provided by user images. Existing methods often suffer from overfitting issues, where the dominant presence of inverted concepts leads to the absence of other desired concepts. We propose a method that guides the inversion process towards the core distribution for compositional embeddings.
  arXiv  Detail & Related papers  (2023-12-13T10:57:46Z)
• Phasic Content Fusing Diffusion Model with Directional Distribution Consistency for Few-Shot Model Adaption [73.98706049140098]
  We propose a novel phasic content fusing few-shot diffusion model with directional distribution consistency loss. Specifically, we design a phasic training strategy with phasic content fusion to help our model learn content and style information when t is large. Finally, we propose a cross-domain structure guidance strategy that enhances structure consistency during domain adaptation.
  arXiv  Detail & Related papers  (2023-09-07T14:14:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.