Multimodal Pragmatic Jailbreak on Text-to-image Models

• URL: http://arxiv.org/abs/2409.19149v1
• Date: Fri, 27 Sep 2024 21:23:46 GMT
• Title: Multimodal Pragmatic Jailbreak on Text-to-image Models
• Authors: Tong Liu, Zhixin Lai, Gengyuan Zhang, Philip Torr, Vera Demberg, Volker Tresp, Jindong Gu,
• Abstract summary: This work introduces a novel type of jailbreak, which triggers T2I models to generate the image with visual text. We benchmark nine representative T2I models, including two close-source commercial models. All tested models suffer from such type of jailbreak, with rates of unsafe generation ranging from 8% to 74%.
• Score: 43.67831238116829
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Diffusion models have recently achieved remarkable advancements in terms of image quality and fidelity to textual prompts. Concurrently, the safety of such generative models has become an area of growing concern. This work introduces a novel type of jailbreak, which triggers T2I models to generate the image with visual text, where the image and the text, although considered to be safe in isolation, combine to form unsafe content. To systematically explore this phenomenon, we propose a dataset to evaluate the current diffusion-based text-to-image (T2I) models under such jailbreak. We benchmark nine representative T2I models, including two close-source commercial models. Experimental results reveal a concerning tendency to produce unsafe content: all tested models suffer from such type of jailbreak, with rates of unsafe generation ranging from 8\% to 74\%. In real-world scenarios, various filters such as keyword blocklists, customized prompt filters, and NSFW image filters, are commonly employed to mitigate these risks. We evaluate the effectiveness of such filters against our jailbreak and found that, while current classifiers may be effective for single modality detection, they fail to work against our jailbreak. Our work provides a foundation for further development towards more secure and reliable T2I models.

Related papers

• ShieldDiff: Suppressing Sexual Content Generation from Diffusion Models through Reinforcement Learning [7.099258248662009]
  There is a potential risk that text-to-image (T2I) model can generate unsafe images with uncomfortable contents. In our work, we focus on eliminating the NSFW (not safe for work) content generation from T2I model. We propose a customized reward function consisting of the CLIP (Contrastive Language-Image Pre-training) and nudity rewards to prune the nudity contents.
  arXiv  Detail & Related papers  (2024-10-04T19:37:56Z)
• Chain-of-Jailbreak Attack for Image Generation Models via Editing Step by Step [62.82566977845765]
  We introduce a novel jailbreaking method called Chain-of-Jailbreak (CoJ) attack, which compromises image generation models through a step-by-step editing process. Our CoJ attack method can successfully bypass the safeguards of models for over 60% cases. We also propose an effective prompting-based method, Think Twice Prompting, that can successfully defend over 95% of CoJ attack.
  arXiv  Detail & Related papers  (2024-10-04T19:04:43Z)
• RT-Attack: Jailbreaking Text-to-Image Models via Random Token [24.61198605177661]
  We introduce a two-stage query-based black-box attack method utilizing random search. In the first stage, we establish a preliminary prompt by maximizing the semantic similarity between the adversarial and target harmful prompts. In the second stage, we use this initial prompt to refine our approach, creating a detailed adversarial prompt aimed at jailbreaking.
  arXiv  Detail & Related papers  (2024-08-25T17:33:40Z)
• Perception-guided Jailbreak against Text-to-Image Models [18.825079959947857]
  We propose an LLM-driven perception-guided jailbreak method, termed PGJ. It is a black-box jailbreak method that requires no specific T2I model (model-free) and generates highly natural attack prompts. The experiments conducted on six open-source models and commercial online services with thousands of prompts have verified the effectiveness of PGJ.
  arXiv  Detail & Related papers  (2024-08-20T13:40:25Z)
• Six-CD: Benchmarking Concept Removals for Benign Text-to-image Diffusion Models [58.74606272936636]
  Text-to-image (T2I) diffusion models have shown exceptional capabilities in generating images that closely correspond to textual prompts. The models could be exploited for malicious purposes, such as generating images with violence or nudity, or creating unauthorized portraits of public figures in inappropriate contexts. concept removal methods have been proposed to modify diffusion models to prevent the generation of malicious and unwanted concepts.
  arXiv  Detail & Related papers  (2024-06-21T03:58:44Z)
• ART: Automatic Red-teaming for Text-to-Image Models to Protect Benign Users [18.3621509910395]
  We propose a novel Automatic Red-Teaming framework, ART, to evaluate the safety risks of text-to-image models. With our comprehensive experiments, we reveal the toxicity of the popular open-source text-to-image models. We also introduce three large-scale red-teaming datasets for studying the safety risks associated with text-to-image models.
  arXiv  Detail & Related papers  (2024-05-24T07:44:27Z)
• Latent Guard: a Safety Framework for Text-to-image Generation [64.49596711025993]
  Existing safety measures are either based on text blacklists, which can be easily circumvented, or harmful content classification. We propose Latent Guard, a framework designed to improve safety measures in text-to-image generation. Inspired by blacklist-based approaches, Latent Guard learns a latent space on top of the T2I model's text encoder, where it is possible to check the presence of harmful concepts.
  arXiv  Detail & Related papers  (2024-04-11T17:59:52Z)
• Jailbreaking Prompt Attack: A Controllable Adversarial Attack against Diffusion Models [10.70975463369742]
  We present the Jailbreaking Prompt Attack (JPA) JPA searches for the target malicious concepts in the text embedding space using a group of antonyms. A prefix prompt is optimized in the discrete vocabulary space to align malicious concepts semantically in the text embedding space.
  arXiv  Detail & Related papers  (2024-04-02T09:49:35Z)
• Ring-A-Bell! How Reliable are Concept Removal Methods for Diffusion Models? [52.238883592674696]
  Ring-A-Bell is a model-agnostic red-teaming tool for T2I diffusion models. It identifies problematic prompts for diffusion models with the corresponding generation of inappropriate content. Our results show that Ring-A-Bell, by manipulating safe prompting benchmarks, can transform prompts that were originally regarded as safe to evade existing safety mechanisms.
  arXiv  Detail & Related papers  (2023-10-16T02:11:20Z)
• Prompting4Debugging: Red-Teaming Text-to-Image Diffusion Models by Finding Problematic Prompts [63.61248884015162]
  Text-to-image diffusion models have shown remarkable ability in high-quality content generation. This work proposes Prompting4 Debugging (P4D) as a tool that automatically finds problematic prompts for diffusion models. Our result shows that around half of prompts in existing safe prompting benchmarks which were originally considered "safe" can actually be manipulated to bypass many deployed safety mechanisms.
  arXiv  Detail & Related papers  (2023-09-12T11:19:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.