Red-Teaming the Stable Diffusion Safety Filter

• URL: http://arxiv.org/abs/2210.04610v5
• Date: Thu, 10 Nov 2022 10:07:37 GMT
• Title: Red-Teaming the Stable Diffusion Safety Filter
• Abstract summary: We show that it is easy to generate disturbing content that bypasses the safety filter. We argue safety measures in future model releases should strive to be fully open and properly documented.
• Score: 5.683172456953383
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Stable Diffusion is a recent open-source image generation model comparable to proprietary models such as DALLE, Imagen, or Parti. Stable Diffusion comes with a safety filter that aims to prevent generating explicit images. Unfortunately, the filter is obfuscated and poorly documented. This makes it hard for users to prevent misuse in their applications, and to understand the filter's limitations and improve it. We first show that it is easy to generate disturbing content that bypasses the safety filter. We then reverse-engineer the filter and find that while it aims to prevent sexual content, it ignores violence, gore, and other similarly disturbing content. Based on our analysis, we argue safety measures in future model releases should strive to be fully open and properly documented to stimulate security contributions from the community.

Related papers

• DiSCO: Defending text-to-image generation through distribution-guided contrastive prompt optimization [65.89030524420933]
  DiSCO is a zero-shot, strictly black-box defense that operates entirely at the prompt level as a plug-and-play module.<n>We demonstrate that DiSCO consistently enhances the safety of both undefended and defended models.<n>As a black-box, architecture-agnostic module, DiSCO can be readily applied to any text-to-image system.
  arXiv  Detail & Related papers  (2026-08-17T19:13:55Z)
• Disciplined Diffusion: Text-to-Image Diffusion Model against NSFW Generation [9.862349752373577]
  Text-to-image (T2I) diffusion models have the ability to build high-quality pictures from text prompts.<n>They pose safety concerns because they can generate offensive or disturbing imagery when provided with harmful inputs.<n>We propose Disciplined Diffusion (DDiffusion), a novel robust text-to-image diffusion that counters Not Safe For Work (NSFW) generation.
  arXiv  Detail & Related papers  (2026-05-01T21:33:06Z)
• Multimodal Prompt Decoupling Attack on the Safety Filters in Text-to-Image Models [73.43013217318965]
  Multimodal Prompt Decoupling Attack (MPDA)<n>MPDA uses image modality to separate the harmful semantic components of the original unsafe prompt.<n>Visual language model generates image captions to ensure semantic consistency between the generated NSFW images and the original unsafe prompts.
  arXiv  Detail & Related papers  (2025-09-21T11:22:32Z)
• CROPS: Model-Agnostic Training-Free Framework for Safe Image Synthesis with Latent Diffusion Models [13.799517170191919]
  Recent research has shown that safety checkers have vulnerabilities against adversarial attacks, allowing them to generate Not Safe For Work (NSFW) images. We propose CROPS, a model-agnostic framework that easily defends against adversarial attacks generating NSFW images without requiring additional training.
  arXiv  Detail & Related papers  (2025-01-09T16:43:21Z)
• Safety Alignment Backfires: Preventing the Re-emergence of Suppressed Concepts in Fine-tuned Text-to-Image Diffusion Models [57.16056181201623]
  Fine-tuning text-to-image diffusion models can inadvertently undo safety measures, causing models to relearn harmful concepts. We present a novel but immediate solution called Modular LoRA, which involves training Safety Low-Rank Adaptation modules separately from Fine-Tuning LoRA components. This method effectively prevents the re-learning of harmful content without compromising the model's performance on new tasks.
  arXiv  Detail & Related papers  (2024-11-30T04:37:38Z)
• DiffGuard: Text-Based Safety Checker for Diffusion Models [1.4375779579519912]
  This paper presents a novel text-based safety filter that outperforms existing solutions. Our research is driven by the critical need to address the misuse of AI-generated content, especially in the context of information warfare.
  arXiv  Detail & Related papers  (2024-11-25T21:47:02Z)
• Safety Without Semantic Disruptions: Editing-free Safe Image Generation via Context-preserving Dual Latent Reconstruction [49.60774626839712]
  Training multimodal generative models can expose users to harmful, unsafe and controversial or culturally-inappropriate outputs. We propose a modular, dynamic solution that leverages safety-context embeddings and a dual reconstruction process to generate safer images. We achieve state-of-the-art results on safe image generation benchmarks, while offering controllable variation of model safety.
  arXiv  Detail & Related papers  (2024-11-21T09:47:13Z)
• AdvI2I: Adversarial Image Attack on Image-to-Image Diffusion models [20.37481116837779]
  AdvI2I is a novel framework that manipulates input images to induce diffusion models to generate NSFW content. By optimizing a generator to craft adversarial images, AdvI2I circumvents existing defense mechanisms. We show that both AdvI2I and AdvI2I-Adaptive can effectively bypass current safeguards.
  arXiv  Detail & Related papers  (2024-10-28T19:15:06Z)
• SAFREE: Training-Free and Adaptive Guard for Safe Text-to-Image And Video Generation [65.30207993362595]
  Unlearning/editing-based methods for safe generation remove harmful concepts from models but face several challenges. We propose SAFREE, a training-free approach for safe T2I and T2V. We detect a subspace corresponding to a set of toxic concepts in the text embedding space and steer prompt embeddings away from this subspace.
  arXiv  Detail & Related papers  (2024-10-16T17:32:23Z)
• AquaLoRA: Toward White-box Protection for Customized Stable Diffusion Models via Watermark LoRA [67.68750063537482]
  Diffusion models have achieved remarkable success in generating high-quality images. Recent works aim to let SD models output watermarked content for post-hoc forensics. We propose textttmethod as the first implementation under this scenario.
  arXiv  Detail & Related papers  (2024-05-18T01:25:47Z)
• Gaussian Shading: Provable Performance-Lossless Image Watermarking for Diffusion Models [71.13610023354967]
  Copyright protection and inappropriate content generation pose challenges for the practical implementation of diffusion models. We propose a diffusion model watermarking technique that is both performance-lossless and training-free.
  arXiv  Detail & Related papers  (2024-04-07T13:30:10Z)
• Divide-and-Conquer Attack: Harnessing the Power of LLM to Bypass Safety Filters of Text-to-Image Models [1.5408065585641535]
  We introduce the Divide-and-Conquer Attack to circumvent the safety filters of state-of-the-art TTI models. We design attack helper prompts that effectively guide LLMs to break down an unethical drawing intent into benign descriptions. Our findings have more severe security implications than methods of manual crafting or iterative TTI model querying.
  arXiv  Detail & Related papers  (2023-12-12T10:04:43Z)
• Ring-A-Bell! How Reliable are Concept Removal Methods for Diffusion Models? [52.238883592674696]
  Ring-A-Bell is a model-agnostic red-teaming tool for T2I diffusion models. It identifies problematic prompts for diffusion models with the corresponding generation of inappropriate content. Our results show that Ring-A-Bell, by manipulating safe prompting benchmarks, can transform prompts that were originally regarded as safe to evade existing safety mechanisms.
  arXiv  Detail & Related papers  (2023-10-16T02:11:20Z)
• SneakyPrompt: Jailbreaking Text-to-image Generative Models [20.645304189835944]
  We propose SneakyPrompt, the first automated attack framework, to jailbreak text-to-image generative models. Given a prompt that is blocked by a safety filter, SneakyPrompt repeatedly queries the text-to-image generative model and strategically perturbs tokens in the prompt based on the query results to bypass the safety filter. Our evaluation shows that SneakyPrompt not only successfully generates NSFW images, but also outperforms existing text adversarial attacks when extended to jailbreak text-to-image generative models.
  arXiv  Detail & Related papers  (2023-05-20T03:41:45Z)
• Benchmarking Robustness to Adversarial Image Obfuscations [22.784762155781436]
  Malicious actors may obfuscate policy violating images to prevent machine learning models from reaching the correct decision. This benchmark, based on ImageNet, simulates the type of obfuscations created by malicious actors.
  arXiv  Detail & Related papers  (2023-01-30T15:36:44Z)
• Filter Grafting for Deep Neural Networks: Reason, Method, and Cultivation [86.91324735966766]
  Filter is the key component in modern convolutional neural networks (CNNs) In this paper, we introduce filter grafting (textbfMethod) to achieve this goal. We develop a novel criterion to measure the information of filters and an adaptive weighting strategy to balance the grafted information among networks.
  arXiv  Detail & Related papers  (2020-04-26T08:36:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.