Every Picture Tells a Dangerous Story: Memory-Augmented Multi-Agent Jailbreak Attacks on VLMs
- URL: http://arxiv.org/abs/2604.12616v1
- Date: Tue, 14 Apr 2026 11:44:59 GMT
- Title: Every Picture Tells a Dangerous Story: Memory-Augmented Multi-Agent Jailbreak Attacks on VLMs
- Abstract summary: multimodal jailbreak strategies fail to engage with the complex semantic structures intrinsic to visual data.<n>We introduce textbfMemJack, a framework that explicitly leverages visual semantics to orchestrate automated jailbreak attacks.<n>MemJack achieves a 71.48% ASR against Qwen3-VL-Plus, scaling to 90% under extended budgets.
- Score: 19.511422290404138
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: The rapid evolution of Vision-Language Models (VLMs) has catalyzed unprecedented capabilities in artificial intelligence; however, this continuous modal expansion has inadvertently exposed a vastly broadened and unconstrained adversarial attack surface. Current multimodal jailbreak strategies primarily focus on surface-level pixel perturbations and typographic attacks or harmful images; however, they fail to engage with the complex semantic structures intrinsic to visual data. This leaves the vast semantic attack surface of original, natural images largely unscrutinized. Driven by the need to expose these deep-seated semantic vulnerabilities, we introduce \textbf{MemJack}, a \textbf{MEM}ory-augmented multi-agent \textbf{JA}ilbreak atta\textbf{CK} framework that explicitly leverages visual semantics to orchestrate automated jailbreak attacks. MemJack employs coordinated multi-agent cooperation to dynamically map visual entities to malicious intents, generate adversarial prompts via multi-angle visual-semantic camouflage, and utilize an Iterative Nullspace Projection (INLP) geometric filter to bypass premature latent space refusals. By accumulating and transferring successful strategies through a persistent Multimodal Experience Memory, MemJack maintains highly coherent extended multi-turn jailbreak attack interactions across different images, thereby improving the attack success rate (ASR) on new images. Extensive empirical evaluations across full, unmodified COCO val2017 images demonstrate that MemJack achieves a 71.48\% ASR against Qwen3-VL-Plus, scaling to 90\% under extended budgets. Furthermore, to catalyze future defensive alignment research, we will release \textbf{MemJack-Bench}, a comprehensive dataset comprising over 113,000 interactive multimodal jailbreak attack trajectories, establishing a vital foundation for developing inherently robust VLMs.
Related papers
- A Self-Evolving Multi-Agent Framework Defense against LLM Jailbreak Attacks [53.12252890906191]
Large language models (LLMs) remain vulnerable to jailbreak attacks.<n>We propose a self-evolving test-time defense built around a persistent, cross-interaction rule memory.
arXiv Detail & Related papers (2026-08-26T16:52:06Z) - Do Agents Dream of False Memories? Black-box Visual Attacks on Long-term Memory in Multimodal AI Agents [10.17267846044351]
We show that unconditional trust in visual data creates a critical vulnerability.<n>We propose Lucid, a black-box adversarial framework that compromises multimodal memory pipelines.<n>Lucid crafts imperceptible perturbations to enable two failure modes based on the availability of historical context.
arXiv Detail & Related papers (2026-07-17T06:05:17Z) - Robustness of Vision Language Models Against Split-Image Harmful Input Attacks [4.937150501683971]
Vision-Language Models (VLMs) are now a core part of modern AI.<n>Recent work proposed several visual jailbreak attacks using single/ holistic images.<n>We introduce novel split-image visual jailbreak attacks (SIVA) that exploit this misalignment.
arXiv Detail & Related papers (2026-02-08T21:52:42Z) - SEMA: Simple yet Effective Learning for Multi-Turn Jailbreak Attacks [53.97948802255959]
We propose a framework that trains a multi-turn attacker without relying on any existing strategies or external data.<n>Prefilling self-tuning enables usable rollouts by fine-tuning on non-refusal, well-structured, multi-turn adversarial prompts.<n>We anchor harmful intent in multi-turn jailbreaks via an intent-drift-aware reward that combines intent alignment, compliance risk, and level of detail.
arXiv Detail & Related papers (2026-02-06T16:44:57Z) - Odysseus: Jailbreaking Commercial Multimodal LLM-integrated Systems via Dual Steganography [77.44136793431893]
We propose a novel jailbreak paradigm that introduces dual steganography to covertly embed malicious queries into benign-looking images.<n>Our Odysseus successfully jailbreaks several pioneering and realistic MLLM-integrated systems, achieving up to 99% attack success rate.
arXiv Detail & Related papers (2025-12-23T08:53:36Z) - RAM++: Robust Representation Learning via Adaptive Mask for All-in-One Image Restoration [94.49712266736141]
RAM++ is a two-stage framework for all-in-one image restoration.<n>It integrates high-level semantic understanding with low-level texture generation.<n>It addresses the limitations of existing degradation-oriented methods in extreme scenarios.
arXiv Detail & Related papers (2025-09-15T15:24:15Z) - GRAF: Multi-turn Jailbreaking via Global Refinement and Active Fabrication [55.63412213263305]
Large Language Models pose notable safety risks due to potential misuse for malicious purposes.<n>We propose a novel multi-turn jailbreaking method that globally refines the attack trajectory at each interaction.<n>In addition, we actively fabricate model responses to suppress safety-related warnings, thereby increasing the likelihood of eliciting harmful outputs.
arXiv Detail & Related papers (2025-06-22T03:15:05Z) - Implicit Jailbreak Attacks via Cross-Modal Information Concealment on Vision-Language Models [20.99874786089634]
Previous jailbreak attacks often inject malicious instructions from text into less aligned modalities, such as vision.<n>We propose a novel implicit jailbreak framework termed IJA that stealthily embeds malicious instructions into images via at least significant bit steganography.<n>On commercial models like GPT-4o and Gemini-1.5 Pro, our method achieves attack success rates of over 90% using an average of only 3 queries.
arXiv Detail & Related papers (2025-05-22T09:34:47Z) - Securing Vision-Language Models with a Robust Encoder Against Jailbreak and Adversarial Attacks [0.0]
Large Vision-Language Models (LVLMs) have significantly advanced AI by excelling in vision-language tasks.
Jailbreak attacks bypass safety protocols and cause the model to generate misleading or harmful responses.
We propose Sim-CLIP+, a novel defense mechanism that adversarially fine-tunes the CLIP vision encoder by leveraging a Siamese architecture.
arXiv Detail & Related papers (2024-09-11T15:39:42Z) - Jailbreak Vision Language Models via Bi-Modal Adversarial Prompt [60.54666043358946]
This paper introduces the Bi-Modal Adversarial Prompt Attack (BAP), which executes jailbreaks by optimizing textual and visual prompts cohesively.
In particular, we utilize a large language model to analyze jailbreak failures and employ chain-of-thought reasoning to refine textual prompts.
arXiv Detail & Related papers (2024-06-06T13:00:42Z) - White-box Multimodal Jailbreaks Against Large Vision-Language Models [61.97578116584653]
We propose a more comprehensive strategy that jointly attacks both text and image modalities to exploit a broader spectrum of vulnerability within Large Vision-Language Models.
Our attack method begins by optimizing an adversarial image prefix from random noise to generate diverse harmful responses in the absence of text input.
An adversarial text suffix is integrated and co-optimized with the adversarial image prefix to maximize the probability of eliciting affirmative responses to various harmful instructions.
arXiv Detail & Related papers (2024-05-28T07:13:30Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.