Orthogonal Concept Erasure for Diffusion Models

• URL: http://arxiv.org/abs/2605.28902v1
• Date: Wed, 27 May 2026 15:58:20 GMT
• Title: Orthogonal Concept Erasure for Diffusion Models
• Abstract summary: Concept erasure has emerged as a promising approach to mitigate undesired or unsafe content in diffusion models.<n>We propose Orthogonal Concept Erasure (OCE), which reformulates editing-based erasure as multiplicative parameter updates.<n>OCE applies layer-wise transformations derived from a closed-form solution to the parameters, enabling precise concept erasure.
• Score: 46.450614977061484
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Concept erasure has emerged as a promising approach to mitigate undesired or unsafe content in diffusion models, yet existing methods still face significant limitations. While training-based methods are effective, their high computational cost limits scalability. Editing-based methods are more efficient and deployment-friendly, yet they struggle to simultaneously achieve precise concept erasure and preserve overall generative capacity. We identify this core limitation of the editing-based methods as reliance on additive parameter updates. Our empirical analysis reveals that concept semantics primarily depend on neuron direction rather than neuron magnitude, while overall generative capacity relies on the angular geometry of neurons. As additive updates inherently entangle direction, magnitude, and angular geometry, they inevitably introduce unintended interference between concept erasure and overall generation performance. To address this, we propose Orthogonal Concept Erasure (OCE), which reformulates editing-based erasure as multiplicative parameter updates from a geometric perspective. Specifically, OCE applies layer-wise orthogonal transformations derived from a closed-form solution to the parameters, enabling precise concept erasure while preserving the neuron magnitude and angular geometry. Furthermore, to address conflicting constraints in multi-concept erasure, OCE introduces a subspace-level objective with structured subspace manipulation, yielding a more effective and scalable erasure. Extensive experiments on single- and multi-concept erasure demonstrate that OCE outperforms existing methods in concept erasure and non-target preservation, erasing up to 100 concepts in 4.3 s. Code: https://github.com/HansSunY/OCE.

Related papers

• EraseSAE: Surgical Concept Erasure in Text-to-Video Diffusion Models via Sparse Autoencoders [103.46709005379591]
  Concept erasure offers a principled remedy by removing unwanted semantics from pretrained models.<n>Existing approaches typically operate at a coarse quality, misaligned with the finegrained, distributed nature of concept representations.<n>We propose EraseSAE, a novel framework that leverages sparse autoencoders to achieve surgical concept erasure.
  arXiv  Detail & Related papers  (2026-09-03T10:23:37Z)
• Empty SPACE: Cross-Attention Sparsity for Concept Erasure in Diffusion Models [9.052494309545569]
  Closed-form concept erasure methods offer a fast alternative to backpropagation-based techniques.<n>We propose SParse cross-Attention-based Concept Erasure (SPACE) to maintain erasure effectiveness in larger models.
  arXiv  Detail & Related papers  (2026-05-11T08:46:29Z)
• Closed-Form Concept Erasure via Double Projections [8.357438653110973]
  Concept erasure is a process of removing unwanted concepts from generative models.<n>We present a linear transformation framework that achieves concept erasure analytically, without any training.<n>Our design yields a deterministic and geometrically interpretable procedure for safe, efficient, and theory-grounded concept removal.
  arXiv  Detail & Related papers  (2026-04-11T05:06:15Z)
• Rethinking Concept Bottleneck Models: From Pitfalls to Solutions [53.84388497227224]
  Concept Bottleneck Models (CBMs) ground predictions in human-understandable concepts.<n>CBM-Suite is a methodological framework to address these challenges.
  arXiv  Detail & Related papers  (2026-03-05T19:37:49Z)
• Zero-Residual Concept Erasure via Progressive Alignment in Text-to-Image Model [15.636542463543066]
  Concept Erasure aims to prevent pretrained text-to-image models from generating content associated with semantic-harmful concepts.<n>Existing methods often result in incomplete erasure due to "non-zero alignment residual"<n>We propose a novel closed-form method ErasePro: it is designed for more complete concept erasure and better preserving overall generative quality.
  arXiv  Detail & Related papers  (2025-08-06T14:19:32Z)
• CURE: Concept Unlearning via Orthogonal Representation Editing in Diffusion Models [7.68494752148263]
  CURE is a training-free concept unlearning framework that operates directly in the weight space of pre-trained diffusion models.<n>The Spectral Eraser identifies and isolates features unique to the undesired concept while preserving safe attributes.<n>CURE achieves a more efficient and thorough removal for targeted artistic styles, objects, identities, or explicit content.
  arXiv  Detail & Related papers  (2025-05-19T03:53:06Z)
• SPEED: Scalable, Precise, and Efficient Concept Erasure for Diffusion Models [56.83154571623655]
  We introduce SPEED, an efficient concept erasure approach that directly edits model parameters.<n>Speedy searches for a null space, a model editing space where parameter updates do not affect non-target concepts.<n>We successfully erase 100 concepts within only 5 seconds.
  arXiv  Detail & Related papers  (2025-03-10T14:40:01Z)
• Fantastic Targets for Concept Erasure in Diffusion Models and Where To Find Them [21.386640828092524]
  Concept erasure has emerged as a promising technique for mitigating the risk of harmful content generation in diffusion models.<n>We propose the Adaptive Guided Erasure (AGE) method, which emphdynamically selects optimal target concepts tailored to each undesirable concept.<n>Results show that AGE significantly outperforms state-of-the-art erasure methods on preserving unrelated concepts while maintaining effective erasure performance.
  arXiv  Detail & Related papers  (2025-01-31T08:17:23Z)
• Reliable and Efficient Concept Erasure of Text-to-Image Diffusion Models [76.39651111467832]
  We introduce Reliable and Efficient Concept Erasure (RECE), a novel approach that modifies the model in 3 seconds without necessitating additional fine-tuning. To mitigate inappropriate content potentially represented by derived embeddings, RECE aligns them with harmless concepts in cross-attention layers. The derivation and erasure of new representation embeddings are conducted iteratively to achieve a thorough erasure of inappropriate concepts.
  arXiv  Detail & Related papers  (2024-07-17T08:04:28Z)
• Separable Multi-Concept Erasure from Diffusion Models [52.51972530398691]
  We propose a Separable Multi-concept Eraser (SepME) to eliminate unsafe concepts from large-scale diffusion models. The latter separates optimizable model weights, making each weight increment correspond to a specific concept erasure. Extensive experiments indicate the efficacy of our approach in eliminating concepts, preserving model performance, and offering flexibility in the erasure or recovery of various concepts.
  arXiv  Detail & Related papers  (2024-02-03T11:10:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.