When Memory Becomes a Vulnerability: Towards Multi-turn Jailbreak Attacks against Text-to-Image Generation Systems

• URL: http://arxiv.org/abs/2504.20376v2
• Date: Thu, 28 Aug 2025 06:34:01 GMT
• Title: When Memory Becomes a Vulnerability: Towards Multi-turn Jailbreak Attacks against Text-to-Image Generation Systems
• Abstract summary: Inception is the first multi-turn jailbreak attack against real-world text-to-image generation systems.<n>We build an emulation T2I system that integrates two-stage safety filters and industrial-grade memory mechanisms.
• Score: 43.251956413566546
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Modern text-to-image (T2I) generation systems (e.g., DALL$\cdot$E 3) exploit the memory mechanism, which captures key information in multi-turn interactions for faithful generation. Despite its practicality, the security analyses of this mechanism have fallen far behind. In this paper, we reveal that it can exacerbate the risk of jailbreak attacks. Previous attacks fuse the unsafe target prompt into one ultimate adversarial prompt, which can be easily detected or lead to the generation of non-unsafe images due to under- or over-detoxification. In contrast, we propose embedding the malice at the inception of the chat session in memory, addressing the above limitations. Specifically, we propose Inception, the first multi-turn jailbreak attack against real-world text-to-image generation systems that explicitly exploits their memory mechanisms. Inception is composed of two key modules: segmentation and recursion. We introduce Segmentation, a semantic-preserving method that generates multi-round prompts. By leveraging NLP analysis techniques, we design policies to decompose a prompt, together with its malicious intent, according to sentence structure, thereby evading safety filters. Recursion further addresses the challenge posed by unsafe sub-prompts that cannot be separated through simple segmentation. It firstly expands the sub-prompt, then invokes segmentation recursively. To facilitate multi-turn adversarial prompts crafting, we build VisionFlow, an emulation T2I system that integrates two-stage safety filters and industrial-grade memory mechanisms. The experiment results show that Inception successfully allures unsafe image generation, surpassing the SOTA by a 20.0\% margin in attack success rate. We also conduct experiments on the real-world commercial T2I generation platforms, further validating the threats of Inception in practice.

Related papers

• TempJail: Temporal Jailbreak Attacks against Image-to-Video Generation Models [6.9498303374340225]
  We investigate three attack scenarios and uncover a temporal vulnerability in I2V systems.<n>We propose TempJail, a novel temporal jailbreak framework for I2V systems.<n>Experiments show that TempJail improves attack success rate over prior state-of-the-art methods by 23.3%.
  arXiv  Detail & Related papers  (2026-08-27T11:15:39Z)
• SafeCA: Safe Cross-Attention Localization and Regulation for Text-to-Video Jailbreak Defense [77.70598852234149]
  Text-to-Video (T2V) generative models are vulnerable to jailbreak attacks in real-world deployment.<n>We propose SafeCA, a feature-level defense mechanism for safe cross-attention localization and regularization.<n> Experimental results show that SafeCA reduces the jailbreak success rate by about 20% on mainstream T2V models.
  arXiv  Detail & Related papers  (2026-08-11T14:01:24Z)
• Dynamic Defense Profiling Enables Cognitive Jailbreak of Text-to-Image Models [9.029672953412787]
  Text-to-Image generative models are vulnerable to adversarial misuse.<n>Most existing jailbreak attacks rely on prompt engineering or black-box optimization.<n>We propose MIND, a cognitive jailbreak framework that reframes adversarial prompt generation as a belief-state inference problem.
  arXiv  Detail & Related papers  (2026-07-20T10:09:30Z)
• PAST2HARM: A Simple Adaptive Past Tense Attack for Jailbreaking Multimodal AI [0.0]
  PAST2HARM is a adaptive jailbreak framework that bypasses refusal training in state of the art multimodal text to image models.<n>We evaluate PAST2HARM on three models Gemini Nano Banana Pro, GPT Image 2, and SD XL achieving attack success rates of 83 percent, 67 percent, and 100 percent in a black box, gradient free setting.<n>The attack elicits diverse harmful outputs, including explicit sexual content, political disinformation, historical denial narratives, hate speech, and self harm glorification.
  arXiv  Detail & Related papers  (2026-05-26T18:16:22Z)
• Every Picture Tells a Dangerous Story: Memory-Augmented Multi-Agent Jailbreak Attacks on VLMs [19.511422290404138]
  multimodal jailbreak strategies fail to engage with the complex semantic structures intrinsic to visual data.<n>We introduce textbfMemJack, a framework that explicitly leverages visual semantics to orchestrate automated jailbreak attacks.<n>MemJack achieves a 71.48% ASR against Qwen3-VL-Plus, scaling to 90% under extended budgets.
  arXiv  Detail & Related papers  (2026-04-14T11:44:59Z)
• Text is All You Need for Vision-Language Model Jailbreaking [45.53821881474976]
  We introduce Text-DJ, a novel jailbreak attack that bypasses the model's safety safeguards.<n>We present all sub-queries and distraction queries to the LVLM simultaneously as a grid of images.<n>We demonstrate that this method successfully circumvents the safety alignment of state-of-the-art LVLMs.
  arXiv  Detail & Related papers  (2026-01-31T00:17:56Z)
• MacPrompt: Maraconic-guided Jailbreak against Text-to-Image Models [21.21184947590066]
  We introduce MacPrompt, a novel black-box and cross-lingual attack that reveals previously overlooked vulnerabilities in T2I safety mechanisms.<n>MacPrompt constructs adversarial prompts by performing cross-lingual character-level recombination of harmful terms.<n>It achieves attack success rates as high as 92% for sex-related content and 90% for violence, effectively breaking even state-of-the-art concept removal defenses.
  arXiv  Detail & Related papers  (2026-01-12T02:16:12Z)
• Multimodal Prompt Decoupling Attack on the Safety Filters in Text-to-Image Models [73.43013217318965]
  Multimodal Prompt Decoupling Attack (MPDA)<n>MPDA uses image modality to separate the harmful semantic components of the original unsafe prompt.<n>Visual language model generates image captions to ensure semantic consistency between the generated NSFW images and the original unsafe prompts.
  arXiv  Detail & Related papers  (2025-09-21T11:22:32Z)
• Anyone Can Jailbreak: Prompt-Based Attacks on LLMs and T2Is [8.214994509812724]
  Large language models (LLMs) and text-to-image (T2I) systems remain vulnerable to prompt-based attacks known as jailbreaks.<n>This paper presents a systems-style investigation into how non-experts reliably circumvent safety mechanisms.<n>We propose a unified taxonomy of prompt-level jailbreak strategies spanning both text-output and T2I models.
  arXiv  Detail & Related papers  (2025-07-29T13:55:23Z)
• PRISM: Programmatic Reasoning with Image Sequence Manipulation for LVLM Jailbreaking [3.718606661938873]
  We propose a novel and effective jailbreak framework inspired by Return-Oriented Programming (ROP) techniques from software security.<n>Our approach decomposes a harmful instruction into a sequence of individually benign visual gadgets.<n>Our findings reveal a critical and underexplored vulnerability that exploits the compositional reasoning abilities of LVLMs.
  arXiv  Detail & Related papers  (2025-07-29T07:13:56Z)
• GhostPrompt: Jailbreaking Text-to-image Generative Models based on Dynamic Optimization [19.44247617251449]
  We introduce GhostPrompt, the first automated jailbreak framework that combines dynamic prompt optimization with multimodal feedback.<n>It achieves state-of-the-art performance, increasing the ShieldLM-7B bypass rate from 12.5% to 99.0%.<n>It generalizes to unseen filters including GPT-4.1 and successfully jailbreaks DALLE 3 to generate NSFW images.
  arXiv  Detail & Related papers  (2025-05-25T05:13:06Z)
• T2V-OptJail: Discrete Prompt Optimization for Text-to-Video Jailbreak Attacks [67.91652526657599]
  We formalize the T2V jailbreak attack as a discrete optimization problem and propose a joint objective-based optimization framework, called T2V-OptJail.<n>We conduct large-scale experiments on several T2V models, covering both open-source models and real commercial closed-source models.<n>The proposed method improves 11.4% and 10.0% over the existing state-of-the-art method in terms of attack success rate.
  arXiv  Detail & Related papers  (2025-05-10T16:04:52Z)
• T2VShield: Model-Agnostic Jailbreak Defense for Text-to-Video Models [88.63040835652902]
  Text to video models are vulnerable to jailbreak attacks, where specially crafted prompts bypass safety mechanisms and lead to the generation of harmful or unsafe content.<n>We propose T2VShield, a comprehensive and model agnostic defense framework designed to protect text to video models from jailbreak threats.<n>Our method systematically analyzes the input, model, and output stages to identify the limitations of existing defenses.
  arXiv  Detail & Related papers  (2025-04-22T01:18:42Z)
• Tit-for-Tat: Safeguarding Large Vision-Language Models Against Jailbreak Attacks via Adversarial Defense [90.71884758066042]
  Large vision-language models (LVLMs) introduce a unique vulnerability: susceptibility to malicious attacks via visual inputs. We propose ESIII (Embedding Security Instructions Into Images), a novel methodology for transforming the visual space from a source of vulnerability into an active defense mechanism.
  arXiv  Detail & Related papers  (2025-03-14T17:39:45Z)
• Foot-In-The-Door: A Multi-turn Jailbreak for LLMs [40.958137601841734]
  A key challenge is jailbreak, where adversarial prompts bypass built-in safeguards to elicit harmful disallowed outputs. Inspired by psychological foot-in-the-door principles, we introduce FITD,a novel multi-turn jailbreak method. Our approach progressively escalates the malicious intent of user queries through intermediate bridge prompts and aligns the model's response by itself to induce toxic responses.
  arXiv  Detail & Related papers  (2025-02-27T06:49:16Z)
• Reasoning-Augmented Conversation for Multi-Turn Jailbreak Attacks on Large Language Models [53.580928907886324]
  Reasoning-Augmented Conversation is a novel multi-turn jailbreak framework. It reformulates harmful queries into benign reasoning tasks. We show that RACE achieves state-of-the-art attack effectiveness in complex conversational scenarios.
  arXiv  Detail & Related papers  (2025-02-16T09:27:44Z)
• MRJ-Agent: An Effective Jailbreak Agent for Multi-Round Dialogue [35.7801861576917]
  Large Language Models (LLMs) demonstrate outstanding performance in their reservoir of knowledge and understanding capabilities. LLMs have been shown to be prone to illegal or unethical reactions when subjected to jailbreak attacks. We propose a novel multi-round dialogue jailbreaking agent, emphasizing the importance of stealthiness in identifying and mitigating potential threats to human values.
  arXiv  Detail & Related papers  (2024-11-06T10:32:09Z)
• HTS-Attack: Heuristic Token Search for Jailbreaking Text-to-Image Models [28.28898114141277]
  Text-to-Image(T2I) models have achieved remarkable success in image generation and editing.<n>These models still have many potential issues, particularly in generating inappropriate or Not-Safe-For-Work(NSFW) content.<n>We propose HTS-Attack, a token search attack method.
  arXiv  Detail & Related papers  (2024-08-25T17:33:40Z)
• Jailbreak Vision Language Models via Bi-Modal Adversarial Prompt [60.54666043358946]
  This paper introduces the Bi-Modal Adversarial Prompt Attack (BAP), which executes jailbreaks by optimizing textual and visual prompts cohesively. In particular, we utilize a large language model to analyze jailbreak failures and employ chain-of-thought reasoning to refine textual prompts.
  arXiv  Detail & Related papers  (2024-06-06T13:00:42Z)
• White-box Multimodal Jailbreaks Against Large Vision-Language Models [61.97578116584653]
  We propose a more comprehensive strategy that jointly attacks both text and image modalities to exploit a broader spectrum of vulnerability within Large Vision-Language Models. Our attack method begins by optimizing an adversarial image prefix from random noise to generate diverse harmful responses in the absence of text input. An adversarial text suffix is integrated and co-optimized with the adversarial image prefix to maximize the probability of eliciting affirmative responses to various harmful instructions.
  arXiv  Detail & Related papers  (2024-05-28T07:13:30Z)
• Automatic Jailbreaking of the Text-to-Image Generative AI Systems [76.9697122883554]
  We study the safety of the commercial T2I generation systems, such as ChatGPT, Copilot, and Gemini, on copyright infringement with naive prompts. We propose a stronger automated jailbreaking pipeline for T2I generation systems, which produces prompts that bypass their safety guards. Our framework successfully jailbreaks the ChatGPT with 11.0% block rate, making it generate copyrighted contents in 76% of the time.
  arXiv  Detail & Related papers  (2024-05-26T13:32:24Z)
• Jailbreaking Prompt Attack: A Controllable Adversarial Attack against Diffusion Models [10.70975463369742]
  We present the Jailbreaking Prompt Attack (JPA) JPA searches for the target malicious concepts in the text embedding space using a group of antonyms. A prefix prompt is optimized in the discrete vocabulary space to align malicious concepts semantically in the text embedding space.
  arXiv  Detail & Related papers  (2024-04-02T09:49:35Z)
• JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models [123.66104233291065]
  Jailbreak attacks cause large language models (LLMs) to generate harmful, unethical, or otherwise objectionable content. evaluating these attacks presents a number of challenges, which the current collection of benchmarks and evaluation techniques do not adequately address. JailbreakBench is an open-sourced benchmark with the following components.
  arXiv  Detail & Related papers  (2024-03-28T02:44:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.