Empty SPACE: Cross-Attention Sparsity for Concept Erasure in Diffusion Models

• URL: http://arxiv.org/abs/2605.10198v1
• Date: Mon, 11 May 2026 08:46:29 GMT
• Title: Empty SPACE: Cross-Attention Sparsity for Concept Erasure in Diffusion Models
• Abstract summary: Closed-form concept erasure methods offer a fast alternative to backpropagation-based techniques.<n>We propose SParse cross-Attention-based Concept Erasure (SPACE) to maintain erasure effectiveness in larger models.
• Score: 9.052494309545569
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Erasing specific concepts from text-to-image diffusion models is essential for avoiding the generation of copyrighted and explicit content. Closed-form concept erasure methods offer a fast alternative to backpropagation-based techniques, but they become less effective when scaling from smaller models such as Stable Diffusion 1.5 to larger models like Stable Diffusion XL. To maintain erasure effectiveness in these larger-scale architectures, we propose SParse cross-Attention-based Concept Erasure (SPACE). SPACE iteratively modifies the cross-attention parameters of a model with a closed-form update that jointly induces sparsity and erases target concepts. By concentrating the concept mapping to a lower-dimensional subspace, SPACE achieves superior erasure efficacy compared to dense baselines. Extensive experimental results show improvements in erasure effectiveness and robustness against adversarial prompts. Furthermore, SPACE achieves 80\%-90\% cross-attention sparsity, reducing the storage requirements for saving the modified parameters by 70\%, demonstrating its memory efficiency.

Related papers

• EraseSAE: Surgical Concept Erasure in Text-to-Video Diffusion Models via Sparse Autoencoders [103.46709005379591]
  Concept erasure offers a principled remedy by removing unwanted semantics from pretrained models.<n>Existing approaches typically operate at a coarse quality, misaligned with the finegrained, distributed nature of concept representations.<n>We propose EraseSAE, a novel framework that leverages sparse autoencoders to achieve surgical concept erasure.
  arXiv  Detail & Related papers  (2026-09-03T10:23:37Z)
• Orthogonal Concept Erasure for Diffusion Models [46.450614977061484]
  Concept erasure has emerged as a promising approach to mitigate undesired or unsafe content in diffusion models.<n>We propose Orthogonal Concept Erasure (OCE), which reformulates editing-based erasure as multiplicative parameter updates.<n>OCE applies layer-wise transformations derived from a closed-form solution to the parameters, enabling precise concept erasure.
  arXiv  Detail & Related papers  (2026-05-27T15:58:20Z)
• FlowErase-RL: Rethinking Concept Erasure as Reward Optimization in Flow Matching Models [47.80885954469943]
  Flow matching models have significantly improved text-to-image generation quality, but also introduce growing safety risks.<n>We propose emphFlowErase-RL, the first GRPO-based framework for concept erasure in flow matching models.<n>We show that our method achieves state-of-the-art erasure performance while maintaining strong image quality and semantic alignment.
  arXiv  Detail & Related papers  (2026-05-19T12:10:09Z)
• Differential Vector Erasure: Unified Training-Free Concept Erasure for Flow Matching Models [49.10620605347065]
  We propose Differential Vector Erasure (DVE), a training-free concept erasure method specifically designed for flow matching models.<n>Our key insight is that semantic concepts are implicitly encoded in the directional structure of the velocity field governing the generative flow.<n>During inference, DVE selectively removes concept-specific components by projecting the velocity field onto the differential direction, enabling precise concept suppression without affecting irrelevant semantics.
  arXiv  Detail & Related papers  (2026-02-01T08:05:45Z)
• Robust Concept Erasure in Diffusion Models: A Theoretical Perspective on Security and Robustness [4.23067546195708]
  textbfSCORE (Secure and Concept-Oriented Robust Erasure) is a novel framework for robust concept removal in diffusion models.<n>SCORE sets a new standard for secure and robust concept erasure in diffusion models.
  arXiv  Detail & Related papers  (2025-09-15T15:05:50Z)
• TRACE: Trajectory-Constrained Concept Erasure in Diffusion Models [0.0]
  Concept erasure aims to remove or suppress specific concept information in a generative model.<n>Trajectory-Constrained Attentional Concept Erasure (TRACE) is a novel method to erase targeted concepts from diffusion models.<n>TRACE achieves state-of-the-art performance, outperforming recent methods such as ANT, EraseAnything, and MACE in terms of removal efficacy and output quality.
  arXiv  Detail & Related papers  (2025-05-29T10:15:22Z)
• SPEED: Scalable, Precise, and Efficient Concept Erasure for Diffusion Models [56.83154571623655]
  We introduce SPEED, an efficient concept erasure approach that directly edits model parameters.<n>Speedy searches for a null space, a model editing space where parameter updates do not affect non-target concepts.<n>We successfully erase 100 concepts within only 5 seconds.
  arXiv  Detail & Related papers  (2025-03-10T14:40:01Z)
• Precise, Fast, and Low-cost Concept Erasure in Value Space: Orthogonal Complement Matters [38.355389084255386]
  We propose a precise, fast, and low-cost concept erasure method called Adaptive Value Decomposer (AdaVD)<n>AdaVD excels in both single and multiple concept erasure, showing 2 to 10 times improvement in prior preservation.
  arXiv  Detail & Related papers  (2024-12-09T01:56:25Z)
• Reliable and Efficient Concept Erasure of Text-to-Image Diffusion Models [76.39651111467832]
  We introduce Reliable and Efficient Concept Erasure (RECE), a novel approach that modifies the model in 3 seconds without necessitating additional fine-tuning. To mitigate inappropriate content potentially represented by derived embeddings, RECE aligns them with harmless concepts in cross-attention layers. The derivation and erasure of new representation embeddings are conducted iteratively to achieve a thorough erasure of inappropriate concepts.
  arXiv  Detail & Related papers  (2024-07-17T08:04:28Z)
• Erasing Undesirable Influence in Diffusion Models [51.225365010401006]
  Diffusion models are highly effective at generating high-quality images but pose risks, such as the unintentional generation of NSFW (not safe for work) content. In this work, we introduce EraseDiff, an algorithm designed to preserve the utility of the diffusion model on retained data while removing the unwanted information associated with the data to be forgotten.
  arXiv  Detail & Related papers  (2024-01-11T09:30:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.