AEGIS: A Mechanism-Guided Defense against Visual Synonym Jailbreaks in Text-to-Image Models

• URL: http://arxiv.org/abs/2607.06120v1
• Date: Tue, 07 Jul 2026 10:27:54 GMT
• Title: AEGIS: A Mechanism-Guided Defense against Visual Synonym Jailbreaks in Text-to-Image Models
• Abstract summary: AEGIS is an inference-time defense that applies similarity-aware repulsion only at the identified vulnerable heads.<n>It preserves benign fidelity, avoids suppressing hard-negative concepts, and transfers to SD 2.1 and FLUX.1 after re-identifying the critical heads for each backbone.
• Score: 22.925421768193956
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Text-to-image diffusion models have achieved high visual fidelity and broad adoption, but remain vulnerable to safety violations when adversaries exploit them to synthesize illicit content. Existing alignment paradigms, from input sanitization to structural feature pruning, are largely organized around unsafe concepts explicitly exposed during filtering, editing, or localization. This leaves a blind spot for visual synonym attacks (VSA), a jailbreak where benign-looking prompts elicit prohibited imagery through implicit visual associations. As a result, current defenses face a safety-utility dilemma: they may either under-mitigate VSA threats or over-suppress visually similar benign concepts. The core challenge is that VSA hides the unsafe target at the textual surface while revealing it through generation-time visual-semantic convergence. In this work, we therefore shift from static suppression of pre-specified unsafe concepts to dynamic tracing of how unsafe semantics emerge during generation. Our mechanistic analysis shows that VSA and explicit unsafe prompts converge through sparse semantic-injecting attention heads, which serve as inference-time bottlenecks for prohibited visual semantics. Based on this insight, we propose AEGIS (Adaptive Evasion Guard via Identification and Steering), an inference-time defense that applies similarity-aware repulsion only at the identified vulnerable heads. Evaluated against 16 baselines, AEGIS improves both safety and utility. On SD 1.4, it reduces ASR to $\mathbf{0.00}/\mathbf{0.03}$ for in-domain violence/nudity VSA and achieves ASRs $\le \mathbf{0.09}$ on out-of-domain explicit and adversarial attacks. It preserves benign fidelity, avoids suppressing hard-negative concepts, and transfers to SD 2.1 and FLUX.1 after re-identifying the critical heads for each backbone.

Related papers

• SafeCA: Safe Cross-Attention Localization and Regulation for Text-to-Video Jailbreak Defense [77.70598852234149]
  Text-to-Video (T2V) generative models are vulnerable to jailbreak attacks in real-world deployment.<n>We propose SafeCA, a feature-level defense mechanism for safe cross-attention localization and regularization.<n> Experimental results show that SafeCA reduces the jailbreak success rate by about 20% on mainstream T2V models.
  arXiv  Detail & Related papers  (2026-08-11T14:01:24Z)
• The Illusion of High Utility in Safety Alignment of Text-to-Image Diffusion Models [49.67749928542536]
  Safety alignment of text-to-image (T2I) diffusion models aims to suppress harmful generations while preserving utility on benign prompts.<n>Recent methods often appear to deliver high safety with high utility, but this conclusion rests largely on coarse global utility metrics.<n>We show that when utility is measured with structured evaluation, this illusion breaks: onA (Text-to-Image Faithfulness evaluation with Question Answering)<n>We propose StructureAware Geometric Regularization (SAGE), a safety alignment objective that explicitly preserves embedding spread and inter-prompt relational structure.
  arXiv  Detail & Related papers  (2026-07-01T04:00:27Z)
• SACE: Concept Erasure at the Semantic Singularity in Visual Autoregressive Models [10.865745057624833]
  We introduce the first scale-aware concept erasure framework (SACE) for visual autoregressive ( VAR) models.<n>By strictly confining interventions to the first scale, our approach couples an Entropy-Regularized Erasure Objective to prevent high-entropy sampling degeneration.
  arXiv  Detail & Related papers  (2026-06-14T13:39:00Z)
• Making MLLMs Blind: Adversarial Smuggling Attacks in MLLM Content Moderation [54.38271718421492]
  Adversarial smuggling attacks exploit the Human-AI capability gap.<n>It encodes harmful content into human-readable visual formats that remain AI-unreadable.<n>We conduct a preliminary exploration of mitigation strategies.
  arXiv  Detail & Related papers  (2026-04-08T11:13:16Z)
• Reasoning-Oriented Programming: Chaining Semantic Gadgets to Jailbreak Large Vision Language Models [9.123902853709206]
  Large Vision-Language Models (LVLMs) undergo safety alignment to suppress harmful content.<n>In this paper, we identify a systemic flaw where LVLMs can be induced to synthesize harmful logic from benign premises.<n>Our approach exploits the model's instruction-following capability to orchestrate a semantic collision of benign inputs.
  arXiv  Detail & Related papers  (2026-03-10T06:18:48Z)
• Visual Self-Fulfilling Alignment: Shaping Safety-Oriented Personas via Threat-Related Images [15.316817046650208]
  Multimodal large language models (MLLMs) face safety misalignment, where visual inputs enable harmful outputs.<n>Inspired by the Self-Fulfilling mechanism underlying emergent misalignment, we propose Visual Self-Fulfilling Alignment (VSFA)<n> VSFA fine-tunes vision-language models (VLMs) on neutral VQA tasks constructed around threat-related images, without any safety labels.
  arXiv  Detail & Related papers  (2026-03-09T15:20:53Z)
• The Illusion of Forgetting: Attack Unlearned Diffusion via Initial Latent Variable Optimization [51.835894707552946]
  Unlearning-based defenses claim to purge Not-Safe-For-Work concepts from diffusion models (DMs)<n>We show that unlearning partially disrupts the mapping between linguistic symbols and the underlying knowledge, which remains intact as dormant memories.<n>We propose IVO, a concise and powerful attack framework that reactivates these dormant memories by reconstructing the broken mappings.
  arXiv  Detail & Related papers  (2026-01-30T02:39:51Z)
• Oyster-I: Beyond Refusal -- Constructive Safety Alignment for Responsible Language Models [93.5740266114488]
  Constructive Safety Alignment (CSA) protects against malicious misuse while actively guiding vulnerable users toward safe and helpful results.<n>Oy1 achieves state-of-the-art safety among open models while retaining high general capabilities.<n>We release Oy1, code, and the benchmark to support responsible, user-centered AI.
  arXiv  Detail & Related papers  (2025-09-02T03:04:27Z)
• AdversariaL attacK sAfety aLIgnment(ALKALI): Safeguarding LLMs through GRACE: Geometric Representation-Aware Contrastive Enhancement- Introducing Adversarial Vulnerability Quality Index (AVQI) [21.209413521884297]
  Adversarial threats against LLMs are escalating faster than current defenses can adapt.<n>We introduce ALKALI, the first rigorously curated adversarial benchmark.<n>We introduce GRACE, an alignment framework coupling preference learning with latent space regularization.
  arXiv  Detail & Related papers  (2025-06-10T15:14:17Z)
• SRD: Reinforcement-Learned Semantic Perturbation for Backdoor Defense in VLMs [57.880467106470775]
  Attackers can inject imperceptible perturbations into the training data, causing the model to generate malicious, attacker-controlled captions.<n>We propose Semantic Reward Defense (SRD), a reinforcement learning framework that mitigates backdoor behavior without prior knowledge of triggers.<n>SRD uses a Deep Q-Network to learn policies for applying discrete perturbations to sensitive image regions, aiming to disrupt the activation of malicious pathways.
  arXiv  Detail & Related papers  (2025-06-05T08:22:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.