Erasing Without Collateral Damage: Precise Concept Removal in Diffusion Models

• URL: http://arxiv.org/abs/2607.05274v1
• Date: Mon, 06 Jul 2026 16:21:09 GMT
• Title: Erasing Without Collateral Damage: Precise Concept Removal in Diffusion Models
• Abstract summary: Training-free concept erasure is attractive mechanism for controlling text-to-image diffusion models.<n>But precise erasure often comes at the cost of damaging semantically related non-target concepts.<n>We introduce CARE, a closed-form concept erasure operator that replaces the raw target direction with a kept-subspace-aware direction.
• Score: 10.269961654607108
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Training-free concept erasure is an attractive mechanism for controlling text-to-image diffusion models, but precise erasure often comes at the cost of damaging semantically related non-target concepts. Existing value-space methods remove the component of each cross-attention value along the target concept direction, implicitly treating target identity and shared visual structure as the same signal. We argue that this is the source of much of the collateral damage in prior preservation. We introduce CARE, a closed-form concept erasure operator that replaces the raw target direction with a kept-subspace-aware direction computed from a small bank of retained concept anchors. The resulting edit is applied directly in cross-attention value space, requires no model fine-tuning, and adds only a negligible offline computation. A single shrinkage parameter controls the erase-preserve trade-off. We further show that the operator admits a minimum-disturbance interpretation and, in its projection form, leaves the kept subspace invariant. Experiments under the standard concept-erasure protocol show that our method preserves non-target concepts more faithfully while maintaining competitive erasure across instance, style, and celebrity concepts. Code: https://github.com/parthupman/care

Related papers

• EraseSAE: Surgical Concept Erasure in Text-to-Video Diffusion Models via Sparse Autoencoders [103.46709005379591]
  Concept erasure offers a principled remedy by removing unwanted semantics from pretrained models.<n>Existing approaches typically operate at a coarse quality, misaligned with the finegrained, distributed nature of concept representations.<n>We propose EraseSAE, a novel framework that leverages sparse autoencoders to achieve surgical concept erasure.
  arXiv  Detail & Related papers  (2026-09-03T10:23:37Z)
• To Erase, or Not to Erase: Robust Training-Free Concept Erasure with Preservation aware Adaptive Ranked Subspace Expansion [10.950549540413528]
  Concept erasure techniques edit text-to-image diffusion models to erase undesired targets such as NSFW content or copyrighted styles.<n>Current CETs face a trade-off between erasure and utility: stronger edits erase the target more reliably but degrade utility on non-target concepts.<n>We present Preservation-aware Adaptive Ranked Subspace Expansion (PARSE), a training-free framework for robust concept erasure in latent diffusion models.
  arXiv  Detail & Related papers  (2026-07-26T06:35:31Z)
• Uni-AdaVD: Universal Concept Erasure for Visual Generation via Orthogonal Value Decomposition [62.947473058878565]
  We present Uni-AdaVD, a universal inference-time concept erasure framework for visual generation.<n>Our results suggest that Uni-AdaVD provides an efficient and adaptable safety mechanism for modern visual generative models.
  arXiv  Detail & Related papers  (2026-07-16T03:16:51Z)
• Concept Unlearning via Cross-Attention Activation Projection for Diffusion Models [6.701045531467819]
  Concept unlearning aims to erase a target concept from a pretrained text-to-image diffusion model without retraining.<n>Existing closed-form methods represent the target concept through the text encoder's response to a few short anchor prompts that name it.<n>We propose PURE, a closed-form method that builds the forget and retain bases from per-layer cross-attention activations.
  arXiv  Detail & Related papers  (2026-05-25T12:18:47Z)
• Neighbor-Aware Localized Concept Erasure in Text-to-Image Diffusion Models [11.53473951431438]
  Concept erasure in text-to-image diffusion models seeks to remove undesired concepts while preserving overall generative capability.<n>We propose Neighbor-Aware Localized Concept Erasure (NLCE), a training-free framework designed to better preserve neighboring concepts while removing target concepts.
  arXiv  Detail & Related papers  (2026-03-27T00:58:11Z)
• Differential Vector Erasure: Unified Training-Free Concept Erasure for Flow Matching Models [49.10620605347065]
  We propose Differential Vector Erasure (DVE), a training-free concept erasure method specifically designed for flow matching models.<n>Our key insight is that semantic concepts are implicitly encoded in the directional structure of the velocity field governing the generative flow.<n>During inference, DVE selectively removes concept-specific components by projecting the velocity field onto the differential direction, enabling precise concept suppression without affecting irrelevant semantics.
  arXiv  Detail & Related papers  (2026-02-01T08:05:45Z)
• Preserving Task-Relevant Information Under Linear Concept Removal [19.953631890863498]
  SPLICE eliminates sensitive concepts from representations while preserving their covariance with a target label.<n>It is the unique solution that removes linear concept predictability and maintains target covariance with minimal embedding distortion.<n> Empirically, SPLICE outperforms baselines on benchmarks such as Bias in Bios and Winobias.
  arXiv  Detail & Related papers  (2025-06-12T13:53:29Z)
• SAGE: Exploring the Boundaries of Unsafe Concept Domain with Semantic-Augment Erasing [65.82241040239452]
  Concept erasing finetunes weights to unlearn undesirable concepts.<n>Existing methods treat unsafe concept as a fixed word and repeatedly erase it.<n>We introduce semantic-augment erasing which transforms concept word erasure into concept domain erasure.
  arXiv  Detail & Related papers  (2025-06-11T03:21:24Z)
• CURE: Concept Unlearning via Orthogonal Representation Editing in Diffusion Models [7.68494752148263]
  CURE is a training-free concept unlearning framework that operates directly in the weight space of pre-trained diffusion models.<n>The Spectral Eraser identifies and isolates features unique to the undesired concept while preserving safe attributes.<n>CURE achieves a more efficient and thorough removal for targeted artistic styles, objects, identities, or explicit content.
  arXiv  Detail & Related papers  (2025-05-19T03:53:06Z)
• Concept Replacer: Replacing Sensitive Concepts in Diffusion Models via Precision Localization [48.20360860166279]
  Large-scale diffusion models produce high-quality images but often generate unwanted content, such as sexually explicit or violent content.<n>We propose a novel approach for targeted concept replacing in diffusion models, enabling specific concepts to be removed without affecting non-target areas.<n>Our method introduces a dedicated concept localizer for precisely identifying the target concept during the denoising process, trained with few-shot learning to require minimal labeled data.<n>Within the identified region, we introduce a training-free Dual Prompts Cross-Attention (DPCA) module to substitute the target concept, ensuring minimal disruption to surrounding content.
  arXiv  Detail & Related papers  (2024-12-02T08:05:39Z)
• Reliable and Efficient Concept Erasure of Text-to-Image Diffusion Models [76.39651111467832]
  We introduce Reliable and Efficient Concept Erasure (RECE), a novel approach that modifies the model in 3 seconds without necessitating additional fine-tuning. To mitigate inappropriate content potentially represented by derived embeddings, RECE aligns them with harmless concepts in cross-attention layers. The derivation and erasure of new representation embeddings are conducted iteratively to achieve a thorough erasure of inappropriate concepts.
  arXiv  Detail & Related papers  (2024-07-17T08:04:28Z)
• LEACE: Perfect linear concept erasure in closed form [97.78661458934953]
  Concept erasure aims to remove specified features from an embedding.<n>We introduce LEAst-squares Concept Erasure (LEACE), a closed-form method which provably prevents all linear classifiers from detecting a concept while changing the embedding as little as possible.<n>We apply LEACE to large language models with a novel procedure called "concept scrubbing," which erases target concept information from every layer in the network.
  arXiv  Detail & Related papers  (2023-06-06T16:07:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.