GuardT2I: Defending Text-to-Image Models from Adversarial Prompts

• URL: http://arxiv.org/abs/2403.01446v2
• Date: Wed, 30 Oct 2024 05:56:30 GMT
• Title: GuardT2I: Defending Text-to-Image Models from Adversarial Prompts
• Authors: Yijun Yang, Ruiyuan Gao, Xiao Yang, Jianyuan Zhong, Qiang Xu,
• Abstract summary: GuardT2I is a novel moderation framework that adopts a generative approach to enhance T2I models' robustness against adversarial prompts. Our experiments reveal that GuardT2I outperforms leading commercial solutions like OpenAI-Moderation and Microsoft Azure Moderator.
• Score: 16.317849859000074
• License:
• Abstract: Recent advancements in Text-to-Image (T2I) models have raised significant safety concerns about their potential misuse for generating inappropriate or Not-Safe-For-Work (NSFW) contents, despite existing countermeasures such as NSFW classifiers or model fine-tuning for inappropriate concept removal. Addressing this challenge, our study unveils GuardT2I, a novel moderation framework that adopts a generative approach to enhance T2I models' robustness against adversarial prompts. Instead of making a binary classification, GuardT2I utilizes a Large Language Model (LLM) to conditionally transform text guidance embeddings within the T2I models into natural language for effective adversarial prompt detection, without compromising the models' inherent performance. Our extensive experiments reveal that GuardT2I outperforms leading commercial solutions like OpenAI-Moderation and Microsoft Azure Moderator by a significant margin across diverse adversarial scenarios. Our framework is available at https://github.com/cure-lab/GuardT2I.

Related papers

• PromptGuard: Soft Prompt-Guided Unsafe Content Moderation for Text-to-Image Models [34.81551119810424]
  Text-to-image (T2I) models have been shown to be vulnerable to misuse, particularly in generating not-safe-for-work (NSFW) content. We present PromptGuard, a novel content moderation technique that draws inspiration from the system prompt mechanism in large language models (LLMs) for safety alignment.
  arXiv  Detail & Related papers  (2025-01-07T05:39:21Z)
• AEIOU: A Unified Defense Framework against NSFW Prompts in Text-to-Image Models [39.11841245506388]
  Malicious users often exploit text-to-image (T2I) models to generate Not-Safe-for-Work (NSFW) images. We introduce AEIOU, a framework that is Adaptable, Efficient, Interpretable, Optimizable, and Unified against NSFW prompts in T2I models.
  arXiv  Detail & Related papers  (2024-12-24T03:17:45Z)
• SafetyDPO: Scalable Safety Alignment for Text-to-Image Generation [68.07258248467309]
  Text-to-image (T2I) models have become widespread, but their limited safety guardrails expose end users to harmful content and potentially allow for model misuse. Current safety measures are typically limited to text-based filtering or concept removal strategies, able to remove just a few concepts from the model's generative capabilities. We introduce SafetyDPO, a method for safety alignment of T2I models through Direct Preference Optimization (DPO) We train safety experts, in the form of low-rank adaptation (LoRA) matrices, able to guide the generation process away from specific safety-related
  arXiv  Detail & Related papers  (2024-12-13T18:59:52Z)
• In-Context Experience Replay Facilitates Safety Red-Teaming of Text-to-Image Diffusion Models [104.94706600050557]
  Text-to-image (T2I) models have shown remarkable progress, but their potential to generate harmful content remains a critical concern in the ML community. We propose ICER, a novel red-teaming framework that generates interpretable and semantic meaningful problematic prompts. Our work provides crucial insights for developing more robust safety mechanisms in T2I systems.
  arXiv  Detail & Related papers  (2024-11-25T04:17:24Z)
• ShieldDiff: Suppressing Sexual Content Generation from Diffusion Models through Reinforcement Learning [7.099258248662009]
  There is a potential risk that text-to-image (T2I) model can generate unsafe images with uncomfortable contents. In our work, we focus on eliminating the NSFW (not safe for work) content generation from T2I model. We propose a customized reward function consisting of the CLIP (Contrastive Language-Image Pre-training) and nudity rewards to prune the nudity contents.
  arXiv  Detail & Related papers  (2024-10-04T19:37:56Z)
• HTS-Attack: Heuristic Token Search for Jailbreaking Text-to-Image Models [28.28898114141277]
  Text-to-Image(T2I) models have achieved remarkable success in image generation and editing. These models still have many potential issues, particularly in generating inappropriate or Not-Safe-For-Work(NSFW) content. We propose HTS-Attack, a token search attack method.
  arXiv  Detail & Related papers  (2024-08-25T17:33:40Z)
• Direct Unlearning Optimization for Robust and Safe Text-to-Image Models [29.866192834825572]
  Unlearning techniques have been developed to remove the model's ability to generate potentially harmful content. These methods are easily bypassed by adversarial attacks, making them unreliable for ensuring the safety of generated images. We propose Direct Unlearning Optimization (DUO), a novel framework for removing Not Safe For Work (NSFW) content from T2I models.
  arXiv  Detail & Related papers  (2024-07-17T08:19:11Z)
• Latent Guard: a Safety Framework for Text-to-image Generation [64.49596711025993]
  Existing safety measures are either based on text blacklists, which can be easily circumvented, or harmful content classification. We propose Latent Guard, a framework designed to improve safety measures in text-to-image generation. Inspired by blacklist-based approaches, Latent Guard learns a latent space on top of the T2I model's text encoder, where it is possible to check the presence of harmful concepts.
  arXiv  Detail & Related papers  (2024-04-11T17:59:52Z)
• Automated Black-box Prompt Engineering for Personalized Text-to-Image Generation [149.96612254604986]
  PRISM is an algorithm that automatically identifies human-interpretable and transferable prompts. It can effectively generate desired concepts given only black-box access to T2I models. Our experiments demonstrate the versatility and effectiveness of PRISM in generating accurate prompts for objects, styles and images.
  arXiv  Detail & Related papers  (2024-03-28T02:35:53Z)
• Mini-DALLE3: Interactive Text to Image by Prompting Large Language Models [71.49054220807983]
  A prevalent limitation persists in the effective communication with T2I models, such as Stable Diffusion, using natural language descriptions. Inspired by the recently released DALLE3, we revisit the existing T2I systems endeavoring to align human intent and introduce a new task - interactive text to image (iT2I) We present a simple approach that augments LLMs for iT2I with prompting techniques and off-the-shelf T2I models.
  arXiv  Detail & Related papers  (2023-10-11T16:53:40Z)
• FLIRT: Feedback Loop In-context Red Teaming [79.63896510559357]
  We propose an automatic red teaming framework that evaluates a given black-box model and exposes its vulnerabilities. Our framework uses in-context learning in a feedback loop to red team models and trigger them into unsafe content generation.
  arXiv  Detail & Related papers  (2023-08-08T14:03:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.