TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models

• URL: http://arxiv.org/abs/2608.15341v1
• Date: Sat, 15 Aug 2026 17:46:37 GMT
• Title: TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
• Abstract summary: Text-to-image diffusion models can be misused to generate harmful content through adversarial or paraphrased prompts.<n>Existing concept erasure methods often suffer from limited robustness against adversarial prompts.<n>We propose a lightweight Text Alignment framework that fine-tunes only the text encoder while keeping the generative backbone fully frozen.
• Score: 15.202937007632777
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Text-to-image diffusion models can be misused to generate harmful content through adversarial or paraphrased prompts that bypass built-in safety mechanisms. Existing concept erasure methods often suffer from limited robustness against adversarial prompts, degradation of benign generation quality, or reliance on inference-time interventions that introduce persistent computational overhead. To address these limitations, we formulate concept erasure as a domain alignment problem in the text representation space. We propose a lightweight Text Encoder Alignment framework (TEA) that fine-tunes only the text encoder while keeping the generative backbone fully frozen. Given concept--anchor prompt pairs, our method trains a discriminator to distinguish token-level representations of concept-containing prompts from those of safe anchor prompts, while updating the text encoder to make these representations indistinguishable. TEA introduces zero inference-time overhead and requires only a small number of fine-tuning steps, making it highly efficient to deploy at scale. Despite this efficiency, TEA achieves state-of-the-art erasure robustness against black-box and white-box adversarial attacks on Stable Diffusion v1.4, while preserving generation quality on benign prompts. Furthermore, TEA is model-agnostic and achieves the lowest attack success rate on Stable Diffusion v3.5, extending concept erasure to a Rectified Flow Transformer architecture with T5 conditioning where prior methods remain largely unexplored. Code is available at \href{https://github.com/alirezafarashah/TEA.git}{https://github.com/alirezafarashah/TEA.git}

Related papers

• Interpreting Object-Dependent Concept Brittleness in Text-to-Image Diffusion Models [105.53190946111586]
  We term this phenomenon object-dependent concept brittleness.<n>We present an interpretability-oriented framework to audit and minimally correct these failures.<n>We evaluate the proposed framework on style and attribute failure cases across multiple diffusion backbones.
  arXiv  Detail & Related papers  (2026-09-09T09:07:40Z)
• EraseSAE: Surgical Concept Erasure in Text-to-Video Diffusion Models via Sparse Autoencoders [103.46709005379591]
  Concept erasure offers a principled remedy by removing unwanted semantics from pretrained models.<n>Existing approaches typically operate at a coarse quality, misaligned with the finegrained, distributed nature of concept representations.<n>We propose EraseSAE, a novel framework that leverages sparse autoencoders to achieve surgical concept erasure.
  arXiv  Detail & Related papers  (2026-09-03T10:23:37Z)
• The Illusion of High Utility in Safety Alignment of Text-to-Image Diffusion Models [49.67749928542536]
  Safety alignment of text-to-image (T2I) diffusion models aims to suppress harmful generations while preserving utility on benign prompts.<n>Recent methods often appear to deliver high safety with high utility, but this conclusion rests largely on coarse global utility metrics.<n>We show that when utility is measured with structured evaluation, this illusion breaks: onA (Text-to-Image Faithfulness evaluation with Question Answering)<n>We propose StructureAware Geometric Regularization (SAGE), a safety alignment objective that explicitly preserves embedding spread and inter-prompt relational structure.
  arXiv  Detail & Related papers  (2026-07-01T04:00:27Z)
• PRISM: Prior Rectification and Uncertainty-Aware Structure Modeling for Diffusion-Based Text Image Super-Resolution [51.96078493242164]
  PRISM is a single-step diffusion-based Text-SR framework.<n> PRISM achieves state-of-the-art performance with millisecond-level inference.
  arXiv  Detail & Related papers  (2026-05-13T05:31:06Z)
• Disciplined Diffusion: Text-to-Image Diffusion Model against NSFW Generation [9.862349752373577]
  Text-to-image (T2I) diffusion models have the ability to build high-quality pictures from text prompts.<n>They pose safety concerns because they can generate offensive or disturbing imagery when provided with harmful inputs.<n>We propose Disciplined Diffusion (DDiffusion), a novel robust text-to-image diffusion that counters Not Safe For Work (NSFW) generation.
  arXiv  Detail & Related papers  (2026-05-01T21:33:06Z)
• TINA: Text-Free Inversion Attack for Unlearned Text-to-Image Diffusion Models [92.56562999211809]
  A text-centric paradigm equates erasure with severing the text-to-image mapping, ignoring that the underlying visual knowledge related to undesired concepts still persist.<n>We introduce Tina, a novel Text-free INversion Attack, which enforces this visual-only probe by operating under a null-text condition.<n>Our experiments demonstrate that Tina regenerates erased concepts from models treated with state-of-the-art unlearning.
  arXiv  Detail & Related papers  (2026-03-18T15:25:03Z)
• SafeRedir: Prompt Embedding Redirection for Robust Unlearning in Image Generation Models [67.84174763413178]
  We introduce SafeRedir, a lightweight inference-time framework for robust unlearning via prompt embedding redirection.<n>We show that SafeRedir achieves effective unlearning capability, high semantic and perceptual preservation, robust image quality, and enhanced resistance to adversarial attacks.
  arXiv  Detail & Related papers  (2026-01-13T15:01:38Z)
• Deferred Commitment Decoding for Diffusion Language Models with Confidence-Aware Sliding Windows [33.361153168706444]
  We propose Deferred Commitment Decoding (DCD) as a training-free decoding strategy.<n>DCD maintains a confidence-aware sliding window over masked tokens, resolving low-uncertainty tokens early while deferring high-uncertainty tokens until sufficient contextual evidence becomes available.<n>Experiments show that DCD improves generation accuracy by 1.39% with comparable time on average compared to fixed block-based diffusion methods, with the most significant improvement reaching 9.0%.
  arXiv  Detail & Related papers  (2026-01-05T12:57:33Z)
• NSFW-Classifier Guided Prompt Sanitization for Safe Text-to-Image Generation [47.03824997129498]
  "jailbreak" attacks in large language models bypass restrictions through subtle prompt modifications.<n>PromptSan is a novel approach to detoxify harmful prompts without altering model architecture.<n>PromptSan achieves state-of-the-art performance in reducing harmful content generation across multiple metrics.
  arXiv  Detail & Related papers  (2025-06-23T06:17:30Z)
• Concept Steerers: Leveraging K-Sparse Autoencoders for Test-Time Controllable Generations [5.2956273221301835]
  Text-to-image generative models are prone to adversarial attacks and inadvertently generate unsafe, unethical content.<n>We propose a novel framework leveraging k-sparse autoencoders (k-SAEs) to enable efficient and interpretable concept manipulation.<n>Our method yields an improvement of $mathbf20.01%$ in unsafe concept removal, is effective in style manipulation, and is $mathbfsim5$x faster than the current state-of-the-art.
  arXiv  Detail & Related papers  (2025-01-31T11:52:47Z)
• Mitigating Sexual Content Generation via Embedding Distortion in Text-conditioned Diffusion Models [2.2917707112773593]
  Distorting Embedding Space (DES) is a text encoder-based defense mechanism.<n>DES transforms unsafe embeddings, extracted from a text encoder using unsafe prompts, toward carefully calculated safe embedding regions.<n>DES also neutralizes the nudity'' embedding, by aligning it with neutral embedding to enhance robustness against adversarial attacks.
  arXiv  Detail & Related papers  (2025-01-31T04:14:05Z)
• Reliable and Efficient Concept Erasure of Text-to-Image Diffusion Models [76.39651111467832]
  We introduce Reliable and Efficient Concept Erasure (RECE), a novel approach that modifies the model in 3 seconds without necessitating additional fine-tuning. To mitigate inappropriate content potentially represented by derived embeddings, RECE aligns them with harmless concepts in cross-attention layers. The derivation and erasure of new representation embeddings are conducted iteratively to achieve a thorough erasure of inappropriate concepts.
  arXiv  Detail & Related papers  (2024-07-17T08:04:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.