ART: Automatic Red-teaming for Text-to-Image Models to Protect Benign Users

• URL: http://arxiv.org/abs/2405.19360v2
• Date: Mon, 17 Jun 2024 10:00:33 GMT
• Title: ART: Automatic Red-teaming for Text-to-Image Models to Protect Benign Users
• Authors: Guanlin Li, Kangjie Chen, Shudong Zhang, Jie Zhang, Tianwei Zhang,
• Abstract summary: We propose a novel Automatic Red-Teaming framework, ART, to evaluate the safety risks of text-to-image models. With our comprehensive experiments, we reveal the toxicity of the popular open-source text-to-image models. We also introduce three large-scale red-teaming datasets for studying the safety risks associated with text-to-image models.
• Score: 18.3621509910395
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large-scale pre-trained generative models are taking the world by storm, due to their abilities in generating creative content. Meanwhile, safeguards for these generative models are developed, to protect users' rights and safety, most of which are designed for large language models. Existing methods primarily focus on jailbreak and adversarial attacks, which mainly evaluate the model's safety under malicious prompts. Recent work found that manually crafted safe prompts can unintentionally trigger unsafe generations. To further systematically evaluate the safety risks of text-to-image models, we propose a novel Automatic Red-Teaming framework, ART. Our method leverages both vision language model and large language model to establish a connection between unsafe generations and their prompts, thereby more efficiently identifying the model's vulnerabilities. With our comprehensive experiments, we reveal the toxicity of the popular open-source text-to-image models. The experiments also validate the effectiveness, adaptability, and great diversity of ART. Additionally, we introduce three large-scale red-teaming datasets for studying the safety risks associated with text-to-image models. Datasets and models can be found in https://github.com/GuanlinLee/ART.

Related papers

• Latent Guard: a Safety Framework for Text-to-image Generation [64.49596711025993]
  Latent Guard is a framework designed to improve safety measures in text-to-image generation. Inspired by blacklist-based approaches, Latent Guard learns a latent space on top of the T2I model's text encoder. Our proposed framework is composed of a data generation pipeline specific to the task.
  arXiv  Detail & Related papers  (2024-04-11T17:59:52Z)
• Privacy Backdoors: Enhancing Membership Inference through Poisoning Pre-trained Models [112.48136829374741]
  In this paper, we unveil a new vulnerability: the privacy backdoor attack. When a victim fine-tunes a backdoored model, their training data will be leaked at a significantly higher rate than if they had fine-tuned a typical model. Our findings highlight a critical privacy concern within the machine learning community and call for a reevaluation of safety protocols in the use of open-source pre-trained models.
  arXiv  Detail & Related papers  (2024-04-01T16:50:54Z)
• Safe-CLIP: Removing NSFW Concepts from Vision-and-Language Models [42.19184265811366]
  We introduce a novel approach to enhancing the safety of vision-and-language models by diminishing their sensitivity to NSFW (not safe for work) inputs. We show how this can be done by fine-tuning a CLIP model on synthetic data obtained from a large language model trained to convert between safe and unsafe sentences.
  arXiv  Detail & Related papers  (2023-11-27T19:02:17Z)
• SurrogatePrompt: Bypassing the Safety Filter of Text-To-Image Models via Substitution [22.882337899780968]
  We develop and exhibit the first prompt attacks on Midjourney, resulting in the production of abundant NSFW images. Our framework, SurrogatePrompt, systematically generates attack prompts, utilizing large language models, image-to-text, and image-to-image modules. Results disclose an 88% success rate in bypassing Midjourney's proprietary safety filter with our attack prompts.
  arXiv  Detail & Related papers  (2023-09-25T13:20:15Z)
• Prompting4Debugging: Red-Teaming Text-to-Image Diffusion Models by Finding Problematic Prompts [63.61248884015162]
  Text-to-image diffusion models have shown remarkable ability in high-quality content generation. This work proposes Prompting4 Debugging (P4D) as a tool that automatically finds problematic prompts for diffusion models. Our result shows that around half of prompts in existing safe prompting benchmarks which were originally considered "safe" can actually be manipulated to bypass many deployed safety mechanisms.
  arXiv  Detail & Related papers  (2023-09-12T11:19:36Z)
• FLIRT: Feedback Loop In-context Red Teaming [71.38594755628581]
  We propose an automatic red teaming framework that evaluates a given model and exposes its vulnerabilities. Our framework uses in-context learning in a feedback loop to red team models and trigger them into unsafe content generation.
  arXiv  Detail & Related papers  (2023-08-08T14:03:08Z)
• BAGM: A Backdoor Attack for Manipulating Text-to-Image Generative Models [54.19289900203071]
  The rise in popularity of text-to-image generative artificial intelligence has attracted widespread public interest. We demonstrate that this technology can be attacked to generate content that subtly manipulates its users. We propose a Backdoor Attack on text-to-image Generative Models (BAGM) Our attack is the first to target three popular text-to-image generative models across three stages of the generative process.
  arXiv  Detail & Related papers  (2023-07-31T08:34:24Z)
• Forget-Me-Not: Learning to Forget in Text-to-Image Diffusion Models [79.50701155336198]
  textbfForget-Me-Not is designed to safely remove specified IDs, objects, or styles from a well-configured text-to-image model in as little as 30 seconds. We demonstrate that Forget-Me-Not can effectively eliminate targeted concepts while maintaining the model's performance on other concepts. It can also be adapted as a lightweight model patch for Stable Diffusion, allowing for concept manipulation and convenient distribution.
  arXiv  Detail & Related papers  (2023-03-30T17:58:11Z)
• Membership Inference Attacks Against Text-to-image Generation Models [23.39695974954703]
  This paper performs the first privacy analysis of text-to-image generation models through the lens of membership inference. We propose three key intuitions about membership information and design four attack methodologies accordingly. All of the proposed attacks can achieve significant performance, in some cases even close to an accuracy of 1, and thus the corresponding risk is much more severe than that shown by existing membership inference attacks.
  arXiv  Detail & Related papers  (2022-10-03T14:31:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.