RedDiffuser: Red Teaming Vision-Language Models for Toxic Continuation via Reinforced Stable Diffusion

• URL: http://arxiv.org/abs/2503.06223v3
• Date: Sun, 03 Aug 2025 09:52:38 GMT
• Title: RedDiffuser: Red Teaming Vision-Language Models for Toxic Continuation via Reinforced Stable Diffusion
• Abstract summary: Vision-Language Models (VLMs) are vulnerable to jailbreak attacks.<n>In this work, we examine an insidious variant of this threat: toxic continuation.<n>We propose RedDiffuser, which generates natural-looking adversarial images that induce toxic continuations.
• Score: 27.68654681867373
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision-Language Models (VLMs) are vulnerable to jailbreak attacks, where adversaries bypass safety mechanisms to elicit harmful outputs. In this work, we examine an insidious variant of this threat: toxic continuation. Unlike standard jailbreaks that rely solely on malicious instructions, toxic continuation arises when the model is given a malicious input alongside a partial toxic output, resulting in harmful completions. This vulnerability poses a unique challenge in multimodal settings, where even subtle image variations can disproportionately affect the model's response. To this end, we propose RedDiffuser (RedDiff), the first red teaming framework that uses reinforcement learning to fine-tune diffusion models into generating natural-looking adversarial images that induce toxic continuations. RedDiffuser integrates a greedy search procedure for selecting candidate image prompts with reinforcement fine-tuning that jointly promotes toxic output and semantic coherence. Experiments demonstrate that RedDiffuser significantly increases the toxicity rate in LLaVA outputs by 10.69% and 8.91% on the original and hold-out sets, respectively. It also exhibits strong transferability, increasing toxicity rates on Gemini by 5.1% and on LLaMA-Vision by 26.83%. These findings uncover a cross-modal toxicity amplification vulnerability in current VLM alignment, highlighting the need for robust multimodal red teaming. We will release the RedDiffuser codebase to support future research.

Related papers

• OTTER: A Red-Teaming System for Toxicity-Evading Jailbreak Prompt Optimization [1.5140372490736642]
  We show that surface toxicity and adversarial intent can be decoupled by replacing as few as five tokens.<n>We present OTTER, a black-box red-teaming framework requiring only standard API access.<n>We provide the first quantitative analysis of the toxicity--bypass relationship and a per-category breakdown.
  arXiv  Detail & Related papers  (2026-06-19T03:55:08Z)
• Benign Inputs, Harmful Outputs: Cross-Modal Jailbreaking via Distributed Semantic Recomposition [10.177453923096754]
  Multimodal Large Language Models (MLLMs) have recently demonstrated remarkable capabilities in content synthesis and autonomous reasoning.<n>Previous safety guardrails are primarily designed for unimodal textual input interception.<n>We propose Distributed Semantic Recomposition (DSR), a novel cross-modal jailbreak framework.
  arXiv  Detail & Related papers  (2026-06-01T07:49:12Z)
• A Fragile Guardrail: Diffusion LLM's Safety Blessing and Its Failure Mode [51.43498132808724]
  We show that Diffusion large language models (D-LLMs) have intrinsic robustness against jailbreak attacks.<n>We identify a simple yet effective failure mode, termed context nesting, where harmful requests are embedded within structured benign contexts.<n>We show that this simple strategy is sufficient to bypass D-LLMs' safety blessing, achieving state-of-the-art attack success rates.
  arXiv  Detail & Related papers  (2026-01-30T23:08:14Z)
• Evolving Prompts for Toxicity Search in Large Language Models [3.2729350470429783]
  ToxSearch is an evolutionary framework that tests model safety by evolving prompts in a steady-state loop.<n>We observe practically meaningful but attenuated cross-model transfer, with roughly halving toxicity on most targets.<n>These results suggest that small, controllable perturbations are effective vehicles for systematic red-teaming.
  arXiv  Detail & Related papers  (2025-11-16T07:47:31Z)
• Detoxifying Large Language Models via Autoregressive Reward Guided Representation Editing [77.75609817898035]
  Large Language Models (LLMs) have demonstrated impressive performance across various tasks, yet they remain vulnerable to generating toxic content.<n>We propose textscAutoregressive textscReward textscGuided textscRe presentation textscEditing (ARGRE)<n>ARGRE explicitly models toxicity transitions within the latent representation space, enabling stable and precise reward-guided editing.
  arXiv  Detail & Related papers  (2025-09-24T03:40:32Z)
• GenBreak: Red Teaming Text-to-Image Generators Using Large Language Models [65.91565607573786]
  Text-to-image (T2I) models can be misused to generate harmful content, including nudity or violence.<n>Recent research on red-teaming and adversarial attacks against T2I models has notable limitations.<n>We propose GenBreak, a framework that fine-tunes a red-team large language model (LLM) to systematically explore underlying vulnerabilities.
  arXiv  Detail & Related papers  (2025-06-11T09:09:12Z)
• Benign-to-Toxic Jailbreaking: Inducing Harmful Responses from Harmless Prompts [16.04435108299333]
  We propose a new paradigm: Benign-to-Toxic (B2T) jailbreak.<n>Unlike prior work, we optimize adversarial images to induce toxic outputs from benign conditioning.<n>Our method outperforms prior approaches, transfers in black-box settings, and complements text-based jailbreaks.
  arXiv  Detail & Related papers  (2025-05-26T17:27:32Z)
• MIRAGE: Multimodal Immersive Reasoning and Guided Exploration for Red-Team Jailbreak Attacks [85.3303135160762]
  MIRAGE is a novel framework that exploits narrative-driven context and role immersion to circumvent safety mechanisms in Multimodal Large Language Models. It achieves state-of-the-art performance, improving attack success rates by up to 17.5% over the best baselines. We demonstrate that role immersion and structured semantic reconstruction can activate inherent model biases, facilitating the model's spontaneous violation of ethical safeguards.
  arXiv  Detail & Related papers  (2025-03-24T20:38:42Z)
• Towards Invisible Backdoor Attack on Text-to-Image Diffusion Model [70.03122709795122]
  Backdoor attacks targeting text-to-image diffusion models have advanced rapidly. Current backdoor samples often exhibit two key abnormalities compared to benign samples. We propose a novel Invisible Backdoor Attack (IBA) to enhance the stealthiness of backdoor samples.
  arXiv  Detail & Related papers  (2025-03-22T10:41:46Z)
• Reasoning-Augmented Conversation for Multi-Turn Jailbreak Attacks on Large Language Models [53.580928907886324]
  Reasoning-Augmented Conversation is a novel multi-turn jailbreak framework.<n>It reformulates harmful queries into benign reasoning tasks.<n>We show that RACE achieves state-of-the-art attack effectiveness in complex conversational scenarios.
  arXiv  Detail & Related papers  (2025-02-16T09:27:44Z)
• Retrievals Can Be Detrimental: A Contrastive Backdoor Attack Paradigm on Retrieval-Augmented Diffusion Models [37.66349948811172]
  Diffusion models (DMs) have recently demonstrated remarkable generation capability.<n>Recent studies empower DMs with the advanced Retrieval-Augmented Generation (RAG) technique.<n>RAG enhances DMs' generation and generalization ability while significantly reducing model parameters.<n>Despite the great success, RAG may introduce novel security issues that warrant further investigation.
  arXiv  Detail & Related papers  (2025-01-23T02:42:28Z)
• Playing Devil's Advocate: Unmasking Toxicity and Vulnerabilities in Large Vision-Language Models [0.4948270494088624]
  Large Vision-Language Models exhibit vulnerabilities, especially in generating potentially toxic or unsafe responses.<n>Malicious actors can exploit these vulnerabilities to propagate toxic content in an automated (or semi-) manner.<n>This study systematically examines the vulnerabilities of open-source LVLMs, including LLaVA, InstructBLIP, Fuyu, and Qwen.
  arXiv  Detail & Related papers  (2025-01-14T21:27:40Z)
• Risk-Averse Finetuning of Large Language Models [15.147772383812313]
  We propose integrating risk-averse principles into Large Language Models (LLMs) fine-tuning to minimize the occurrence of harmful outputs.<n> Empirical evaluations on sentiment modification and toxicity mitigation tasks demonstrate the efficacy of risk-averse reinforcement learning with human feedback.
  arXiv  Detail & Related papers  (2025-01-12T19:48:21Z)
• Retention Score: Quantifying Jailbreak Risks for Vision Language Models [60.48306899271866]
  Vision-Language Models (VLMs) are integrated with Large Language Models (LLMs) to enhance multi-modal machine learning capabilities. This paper aims to assess the resilience of VLMs against jailbreak attacks that can compromise model safety compliance and result in harmful outputs. To evaluate a VLM's ability to maintain its robustness against adversarial input perturbations, we propose a novel metric called the textbfRetention Score.
  arXiv  Detail & Related papers  (2024-12-23T13:05:51Z)
• PBI-Attack: Prior-Guided Bimodal Interactive Black-Box Jailbreak Attack for Toxicity Maximization [8.819101213981053]
  We propose a Prior-Guided Bimodal Interactive Black-Box Jailbreak Attack for toxicity.<n>Our method begins by extracting malicious features from a harmful corpus using an alternative LVLM.<n>We enhance these features through bidirectional cross-modal interaction optimization.<n>Experiments demonstrate that PBI-Attack outperforms previous state-of-the-art jailbreak methods.
  arXiv  Detail & Related papers  (2024-12-08T11:14:16Z)
• CopyrightShield: Enhancing Diffusion Model Security against Copyright Infringement Attacks [61.06621533874629]
  Diffusion models are vulnerable to copyright infringement attacks, where attackers inject strategically modified non-infringing images into the training set.<n>We first propose a defense framework, CopyrightShield, to defend against the above attack.<n> Experimental results demonstrate that CopyrightShield significantly improves poisoned sample detection performance across two attack scenarios.
  arXiv  Detail & Related papers  (2024-12-02T14:19:44Z)
• The Dark Side of Human Feedback: Poisoning Large Language Models via User Inputs [8.449922248196705]
  We present a subtle yet effective poisoning attack via user-supplied prompts to penetrate alignment training protections. Our attack, even without explicit knowledge about the target LLMs in the black-box setting, subtly alters the reward feedback mechanism. By injecting 1% of these specially crafted prompts into the data, through malicious users, we demonstrate a toxicity score up to two times higher when a specific trigger word is used.
  arXiv  Detail & Related papers  (2024-09-01T17:40:04Z)
• ASTPrompter: Weakly Supervised Automated Language Model Red-Teaming to Identify Low-Perplexity Toxic Prompts [31.481630330369427]
  We propose a reinforcement learning formulation of red-teaming to discover prompts that both elicit toxic outputs from a defender and have low perplexity as scored by that defender. Our policy performs competitively, producing prompts that induce defender toxicity at a rate of 2-23 times higher than baseline across model scales. Our method creates black-box attacks with 5.4-14 times increased toxicity.
  arXiv  Detail & Related papers  (2024-07-12T17:33:34Z)
• Jailbreaking as a Reward Misspecification Problem [80.52431374743998]
  We propose a novel perspective that attributes this vulnerability to reward misspecification during the alignment process.<n>We introduce a metric ReGap to quantify the extent of reward misspecification and demonstrate its effectiveness.<n>We present ReMiss, a system for automated red teaming that generates adversarial prompts in a reward-misspecified space.
  arXiv  Detail & Related papers  (2024-06-20T15:12:27Z)
• Watch the Watcher! Backdoor Attacks on Security-Enhancing Diffusion Models [65.30406788716104]
  This work investigates the vulnerabilities of security-enhancing diffusion models. We demonstrate that these models are highly susceptible to DIFF2, a simple yet effective backdoor attack. Case studies show that DIFF2 can significantly reduce both post-purification and certified accuracy across benchmark datasets and models.
  arXiv  Detail & Related papers  (2024-06-14T02:39:43Z)
• White-box Multimodal Jailbreaks Against Large Vision-Language Models [61.97578116584653]
  We propose a more comprehensive strategy that jointly attacks both text and image modalities to exploit a broader spectrum of vulnerability within Large Vision-Language Models. Our attack method begins by optimizing an adversarial image prefix from random noise to generate diverse harmful responses in the absence of text input. An adversarial text suffix is integrated and co-optimized with the adversarial image prefix to maximize the probability of eliciting affirmative responses to various harmful instructions.
  arXiv  Detail & Related papers  (2024-05-28T07:13:30Z)
• Safeguarding Vision-Language Models Against Patched Visual Prompt Injectors [31.383591942592467]
  Vision-language models (VLMs) offer innovative ways to combine visual and textual data for enhanced understanding and interaction. Patch-based adversarial attack is considered the most realistic threat model in physical vision applications. We introduce SmoothVLM, a defense mechanism rooted in smoothing techniques, to protectVLMs from the threat of patched visual prompt injectors.
  arXiv  Detail & Related papers  (2024-05-17T04:19:19Z)
• Towards Building a Robust Toxicity Predictor [13.162016701556725]
  This paper presents a novel adversarial attack, texttToxicTrap, introducing small word-level perturbations to fool SOTA text classifiers to predict toxic text samples as benign. Two novel goal function designs allow ToxicTrap to identify weaknesses in both multiclass and multilabel toxic language detectors.
  arXiv  Detail & Related papers  (2024-04-09T22:56:05Z)
• RigorLLM: Resilient Guardrails for Large Language Models against Undesired Content [62.685566387625975]
  Current mitigation strategies, while effective, are not resilient under adversarial attacks. This paper introduces Resilient Guardrails for Large Language Models (RigorLLM), a novel framework designed to efficiently moderate harmful and unsafe inputs.
  arXiv  Detail & Related papers  (2024-03-19T07:25:02Z)
• Breaking Free: How to Hack Safety Guardrails in Black-Box Diffusion Models! [52.0855711767075]
  EvoSeed is an evolutionary strategy-based algorithmic framework for generating photo-realistic natural adversarial samples. We employ CMA-ES to optimize the search for an initial seed vector, which, when processed by the Conditional Diffusion Model, results in the natural adversarial sample misclassified by the Model. Experiments show that generated adversarial images are of high image quality, raising concerns about generating harmful content bypassing safety classifiers.
  arXiv  Detail & Related papers  (2024-02-07T09:39:29Z)
• Revealing Vulnerabilities in Stable Diffusion via Targeted Attacks [41.531913152661296]
  We formulate the problem of targeted adversarial attack on Stable Diffusion and propose a framework to generate adversarial prompts. Specifically, we design a gradient-based embedding optimization method to craft reliable adversarial prompts that guide stable diffusion to generate specific images. After obtaining successful adversarial prompts, we reveal the mechanisms that cause the vulnerability of the model.
  arXiv  Detail & Related papers  (2024-01-16T12:15:39Z)
• Forcing Generative Models to Degenerate Ones: The Power of Data Poisoning Attacks [10.732558183444985]
  Malicious actors can covertly exploit large language models (LLMs) vulnerabilities through poisoning attacks aimed at generating undesirable outputs. This paper explores various poisoning techniques to assess their effectiveness across a range of generative tasks. We show that it is possible to successfully poison an LLM during the fine-tuning stage using as little as 1% of the total tuning data samples.
  arXiv  Detail & Related papers  (2023-12-07T23:26:06Z)
• Unveiling the Implicit Toxicity in Large Language Models [77.90933074675543]
  The open-endedness of large language models (LLMs) combined with their impressive capabilities may lead to new safety issues when being exploited for malicious use. We show that LLMs can generate diverse implicit toxic outputs that are exceptionally difficult to detect via simply zero-shot prompting. We propose a reinforcement learning (RL) based attacking method to further induce the implicit toxicity in LLMs.
  arXiv  Detail & Related papers  (2023-11-29T06:42:36Z)
• Adversarial Robustness through the Lens of Causality [105.51753064807014]
  adversarial vulnerability of deep neural networks has attracted significant attention in machine learning. We propose to incorporate causality into mitigating adversarial vulnerability. Our method can be seen as the first attempt to leverage causality for mitigating adversarial vulnerability.
  arXiv  Detail & Related papers  (2021-06-11T06:55:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.