In-Context Experience Replay Facilitates Safety Red-Teaming of Text-to-Image Diffusion Models

• URL: http://arxiv.org/abs/2411.16769v2
• Date: Wed, 12 Feb 2025 06:39:07 GMT
• Title: In-Context Experience Replay Facilitates Safety Red-Teaming of Text-to-Image Diffusion Models
• Authors: Zhi-Yi Chin, Mario Fritz, Pin-Yu Chen, Wei-Chen Chiu,
• Abstract summary: Text-to-image (T2I) models have shown remarkable progress, but their potential to generate harmful content remains a critical concern in the ML community. We propose ICER, a novel red-teaming framework that generates interpretable and semantic meaningful problematic prompts. Our work provides crucial insights for developing more robust safety mechanisms in T2I systems.
• Score: 104.94706600050557
• License:
• Abstract: Text-to-image (T2I) models have shown remarkable progress, but their potential to generate harmful content remains a critical concern in the ML community. While various safety mechanisms have been developed, the field lacks systematic tools for evaluating their effectiveness against real-world misuse scenarios. In this work, we propose ICER, a novel red-teaming framework that leverages Large Language Models (LLMs) and a bandit optimization-based algorithm to generate interpretable and semantic meaningful problematic prompts by learning from past successful red-teaming attempts. Our ICER efficiently probes safety mechanisms across different T2I models without requiring internal access or additional training, making it broadly applicable to deployed systems. Through extensive experiments, we demonstrate that ICER significantly outperforms existing prompt attack methods in identifying model vulnerabilities while maintaining high semantic similarity with intended content. By uncovering that successful jailbreaking instances can systematically facilitate the discovery of new vulnerabilities, our work provides crucial insights for developing more robust safety mechanisms in T2I systems.

Related papers

• How Jailbreak Defenses Work and Ensemble? A Mechanistic Investigation [39.44000290664494]
  Jailbreak attacks, where harmful prompts bypass generative models' built-in safety, raise serious concerns about model vulnerability. This paper systematically examines jailbreak defenses by reframing the standard generation task as a binary classification problem. We identify two key defense mechanisms: safety shift, which increases refusal rates across all queries, and harmfulness discrimination, which improves the model's ability to distinguish between harmful and benign inputs.
  arXiv  Detail & Related papers  (2025-02-20T12:07:40Z)
• Robust-LLaVA: On the Effectiveness of Large-Scale Robust Image Encoders for Multi-modal Large Language Models [26.656858396343726]
  Multi-modal Large Language Models (MLLMs) excel in vision-language tasks but remain vulnerable to visual adversarial perturbations. Existing methods seek to mitigate these risks by applying constrained adversarial fine-tuning to CLIP vision encoders on ImageNet-scale data. We explore an alternative approach of leveraging existing vision classification models that have been adversarially pre-trained on large-scale data.
  arXiv  Detail & Related papers  (2025-02-03T17:59:45Z)
• Safe to Serve: Aligning Instruction-Tuned Models for Safety and Helpfulness [0.0]
  Large language models (LLMs) have demonstrated remarkable capabilities in complex reasoning and text generation. LLMs can inadvertently generate unsafe or biased responses when prompted with problematic inputs. This research addresses the critical challenge of developing language models that generate both helpful and harmless content.
  arXiv  Detail & Related papers  (2024-11-26T06:52:22Z)
• Direct Unlearning Optimization for Robust and Safe Text-to-Image Models [29.866192834825572]
  Unlearning techniques have been developed to remove the model's ability to generate potentially harmful content. These methods are easily bypassed by adversarial attacks, making them unreliable for ensuring the safety of generated images. We propose Direct Unlearning Optimization (DUO), a novel framework for removing Not Safe For Work (NSFW) content from T2I models.
  arXiv  Detail & Related papers  (2024-07-17T08:19:11Z)
• Watch the Watcher! Backdoor Attacks on Security-Enhancing Diffusion Models [65.30406788716104]
  This work investigates the vulnerabilities of security-enhancing diffusion models. We demonstrate that these models are highly susceptible to DIFF2, a simple yet effective backdoor attack. Case studies show that DIFF2 can significantly reduce both post-purification and certified accuracy across benchmark datasets and models.
  arXiv  Detail & Related papers  (2024-06-14T02:39:43Z)
• Ring-A-Bell! How Reliable are Concept Removal Methods for Diffusion Models? [52.238883592674696]
  Ring-A-Bell is a model-agnostic red-teaming tool for T2I diffusion models. It identifies problematic prompts for diffusion models with the corresponding generation of inappropriate content. Our results show that Ring-A-Bell, by manipulating safe prompting benchmarks, can transform prompts that were originally regarded as safe to evade existing safety mechanisms.
  arXiv  Detail & Related papers  (2023-10-16T02:11:20Z)
• Prompting4Debugging: Red-Teaming Text-to-Image Diffusion Models by Finding Problematic Prompts [63.61248884015162]
  Text-to-image diffusion models have shown remarkable ability in high-quality content generation. This work proposes Prompting4 Debugging (P4D) as a tool that automatically finds problematic prompts for diffusion models. Our result shows that around half of prompts in existing safe prompting benchmarks which were originally considered "safe" can actually be manipulated to bypass many deployed safety mechanisms.
  arXiv  Detail & Related papers  (2023-09-12T11:19:36Z)
• FLIRT: Feedback Loop In-context Red Teaming [79.63896510559357]
  We propose an automatic red teaming framework that evaluates a given black-box model and exposes its vulnerabilities. Our framework uses in-context learning in a feedback loop to red team models and trigger them into unsafe content generation.
  arXiv  Detail & Related papers  (2023-08-08T14:03:08Z)
• Self-Destructing Models: Increasing the Costs of Harmful Dual Uses of Foundation Models [103.71308117592963]
  We present an algorithm for training self-destructing models leveraging techniques from meta-learning and adversarial learning. In a small-scale experiment, we show MLAC can largely prevent a BERT-style model from being re-purposed to perform gender identification.
  arXiv  Detail & Related papers  (2022-11-27T21:43:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.