Robust and Generalizable Safety Steering for Text-to-Image Diffusion Transformers

• URL: http://arxiv.org/abs/2605.30049v1
• Date: Thu, 28 May 2026 15:00:18 GMT
• Title: Robust and Generalizable Safety Steering for Text-to-Image Diffusion Transformers
• Abstract summary: We propose SafeDIG, a safety steering framework that formulates DiT safety adaptation as position-aware sparse feature transfer.<n> Experiments on FLUX.1 Dev and Stable Diffusion 3.5 Large show that SafeDIG consistently reduces target-domain and overall unsafe generation rates.
• Score: 28.791649151495008
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Diffusion Transformers have become a powerful backbone for text-to-image generation, but their layered and cross-modal generation process makes safety control fundamentally different from prompt-level filtering or output-level detection. Harmful semantics may be weakly expressed in text representations, progressively bound to visual latents, and finally entangled with rendering dynamics. As a result, safety steering at a fixed layer can be unstable, and a steering mechanism learned from known risks may not transfer reliably to a shifted target risk domain. We propose SafeDIG, a safety steering framework that formulates DiT safety adaptation as position-aware sparse feature transfer. SafeDIG first constructs Sparse Autoencoders over functionally distinct DiT intervention positions and uses robustness-aware pre-training routing to prioritize intervention sites that are expected to remain stable under source-target risk shift. It then separates transferable safety features from domain-specific activation geometry by freezing the SAE encoder as a reusable sparse safety dictionary and adapting only the decoder to the target-domain activation manifold. During inference, SafeDIG combines Blend and Repel operations to steer unsafe activations toward transferred safety manifolds or away from harmful sparse directions. Experiments on FLUX.1 Dev and Stable Diffusion 3.5 Large show that SafeDIG consistently reduces target-domain and overall unsafe generation rates while preserving source-domain safety and image quality.

Related papers

• Pulling The REINS: Training-Free Safety Alignment of Video Diffusion Models via Representation Steering [20.152820762914214]
  We present REINS, a training-free method that aligns video diffusion models at inference time by steering internal representations toward safe generation.<n>Our key finding is that safety-relevant structure is linearly encoded in the hidden-state activations of video diffusion transformers.<n>At inference, adding this direction to hidden states at an intermediate transformer layer redirects generation from harmful content to semantically related safe alternatives, with no weight updates, no concept enumeration, and negligible computational overhead.
  arXiv  Detail & Related papers  (2026-06-15T20:03:09Z)
• Unified Safe In-context Image Generation in Multimodal Diffusion Transformers via Restricting Unsafe Information Flows [20.386952794426833]
  Unified Visual Safety Regulator (UVR) is a training-free safe generation framework that regulates unsafe semantics in generated images.<n>UVR mitigates unsafe generation through unified, targeted attention modulation and explicit restriction of harmful information flow.<n>Experiments show that UVR achieves state-of-the-art safety performance by achieving 91% and 77% erase rate in image synthesis and editing tasks.
  arXiv  Detail & Related papers  (2026-06-05T03:43:04Z)
• Do Models Share Safety Representations? Cross-Model Steering for Safe Visual Generation [52.122731171289665]
  We introduce the first framework for cross-model safety steering.<n>Our pipeline never accesses unsafe data on the target side.<n>We evaluate our approach in text-to-image and text-to-video generation across diverse source-target model pairs.
  arXiv  Detail & Related papers  (2026-06-03T18:00:04Z)
• SafeCtrl: Region-Aware Safety Control for Text-to-Image Diffusion via Detect-Then-Suppress [13.834468340496414]
  We propose SafeCtrl, a Region-Aware safety control framework operating on a Detect-Then-Suppress paradigm.<n>Unlike global safety interventions, SafeCtrl first employs an attention-guided Detect module to precisely localize specific risk regions.<n>We show that SafeCtrl achieves a superior trade-off between safety and fidelity compared to state-of-the-art methods.
  arXiv  Detail & Related papers  (2026-04-05T03:06:17Z)
• SafeRoPE: Risk-specific Head-wise Embedding Rotation for Safe Generation in Rectified Flow Transformers [20.61907377210064]
  We present SafeRoPE, a safe generation framework for transformer-based diffusion models (e.g., MMDiT)<n>We show that unsafe semantics concentrate within interpretable, low-dimensional subspaces at head level, where a finite set of safety-critical heads is responsible for unsafe feature extraction.<n>We then introduce head-wise RoPE perturbations that can suppress unsafe semantics without degrading benign content or image quality.
  arXiv  Detail & Related papers  (2026-04-02T09:37:32Z)
• Safe Transformer: An Explicit Safety Bit For Interpretable And Controllable Alignment [41.47485992177247]
  We propose Safe Transformer, a modular approach that augments pre-trained language models.<n>The safety bit serves as both an interpretable signal of the model's safety classification and a controllable switch.<n>In red-team benchmarks, Safe Transformer achieves near-zero Attack Success Rate.
  arXiv  Detail & Related papers  (2026-03-06T02:54:16Z)
• PoSafeNet: Safe Learning with Poset-Structured Neural Nets [49.854863600271614]
  existing approaches often enforce multiple safety constraints uniformly or via fixed priority orders, leading to infeasibility and brittle behavior.<n>We formalize this setting as poset-structured safety, modeling safety constraints as a partially ordered set and treating safety composition as a structural property of the policy class.<n>Building on this formulation, we propose PoSafeNet, a differentiable neural safety layer that enforces safety via sequential closed-form projection.
  arXiv  Detail & Related papers  (2026-01-29T22:03:32Z)
• SafeRedir: Prompt Embedding Redirection for Robust Unlearning in Image Generation Models [67.84174763413178]
  We introduce SafeRedir, a lightweight inference-time framework for robust unlearning via prompt embedding redirection.<n>We show that SafeRedir achieves effective unlearning capability, high semantic and perceptual preservation, robust image quality, and enhanced resistance to adversarial attacks.
  arXiv  Detail & Related papers  (2026-01-13T15:01:38Z)
• RoboSafe: Safeguarding Embodied Agents via Executable Safety Logic [56.38397499463889]
  Embodied agents powered by vision-language models (VLMs) are increasingly capable of executing complex real-world tasks.<n>However, they remain vulnerable to hazardous instructions that may trigger unsafe behaviors.<n>We propose RoboSafe, a runtime safeguard for embodied agents through executable predicate-based safety logic.
  arXiv  Detail & Related papers  (2025-12-24T15:01:26Z)
• SafeGRPO: Self-Rewarded Multimodal Safety Alignment via Rule-Governed Policy Optimization [79.14563283347773]
  Multimodal large language models (MLLMs) have demonstrated impressive reasoning and instruction-following capabilities.<n>Cross-modal couplings can produce unsafe semantics even when individual inputs are benign.<n>We propose SafeGRPO, a self-rewarded multimodal safety alignment framework.
  arXiv  Detail & Related papers  (2025-11-17T05:09:49Z)
• UpSafe$^\circ$C: Upcycling for Controllable Safety in Large Language Models [67.91151588917396]
  Large Language Models (LLMs) have achieved remarkable progress across a wide range of tasks, but remain vulnerable to safety risks such as harmful content generation and jailbreak attacks.<n>We propose UpSafe$circ$C, a unified framework for enhancing LLM safety through safety-aware upcycling.<n>Our results highlight a new direction for LLM safety: moving from static alignment toward dynamic, modular, and inference-aware control.
  arXiv  Detail & Related papers  (2025-10-02T16:43:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.