Fully Unleashing the Multimodal Attacker: Meta-Adaptive Jailbreaking of Vision-Language Models

• URL: http://arxiv.org/abs/2608.27531v2
• Date: Thu, 03 Sep 2026 10:56:47 GMT
• Title: Fully Unleashing the Multimodal Attacker: Meta-Adaptive Jailbreaking of Vision-Language Models
• Abstract summary: We propose Meta-Adaptive Multimodal Jailbreaking (MAMJ) to optimize the attacker itself along two axes: an attack strategy prompt (ASP) governing attack iteration and attacker model weights determining attack effectiveness.<n>On MM-SafetyBench, MAMJ achieves 81.0%, 78.9%, and 82.3% ASR against GPT-4o, Gemini-3-Pro-adaptive, and Seed 2.0, respectively, outperforming the strongest sample-level baseline by up to 24.1 percentage points.
• Score: 25.860894697929737
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The safety of large vision-language models is increasingly stress-tested by multimodal jailbreaks, yet existing attacks remain largely static at the meta level: template-based attacks freeze the image-text layout, while iterative attacks adapt only the image-text content with fixed attack strategies and frozen attacker parameters. We propose Meta-Adaptive Multimodal Jailbreaking (MAMJ), which instead optimizes the attacker itself along two axes: an attack strategy prompt (ASP) governing attack iteration and attacker model weights determining attack effectiveness. Across groups of multimodal attack trajectories, an LLM-based critique first refines the ASP, after which group-aggregated attack success rate (ASR) rewards update those weights. On MM-SafetyBench, MAMJ achieves 81.0%, 78.9%, and 82.3% ASR against GPT-4o, Gemini-3-Pro-Preview, and Seed 2.0, respectively, outperforming the strongest sample-level baseline by up to 24.1 percentage points. The learned attacker, comprising the optimized ASP and attacker weights, also transfers without retraining to unseen victims and remains effective under representative defenses. These results reveal a systemic vulnerability of frontier VLMs to meta-adaptive jailbreaks and motivate defenses against meta-level adversaries. Code is available at https://github.com/Alibaba-VELLDEPTH/MetaJailbreak-VLM.

Related papers

• A Self-Evolving Multi-Agent Framework Defense against LLM Jailbreak Attacks [53.12252890906191]
  Large language models (LLMs) remain vulnerable to jailbreak attacks.<n>We propose a self-evolving test-time defense built around a persistent, cross-interaction rule memory.
  arXiv  Detail & Related papers  (2026-08-26T16:52:06Z)
• Adaptive Adversaries: A Multi-Turn, Multi-LLM Benchmark for LLM Agent Security [2.3029886838540032]
  LLM-based agents process external content, exposing them to prompt injection and multi-turn manipulation.<n>We present a 21-scenario benchmark for emphadaptive multi-round attacks against memoryless LLM defenders.
  arXiv  Detail & Related papers  (2026-07-20T15:30:38Z)
• Automated jailbreak attack targeting multiple defense strategies [11.710908982928594]
  Large language models (LLMs) have demonstrated remarkable capabilities across a wide range of tasks.<n>However, their safety remains a critical concern due to their susceptibility to adversarial prompt-based attacks.<n>We present UNIATTACK, an adversarial testing framework designed from a defense-oriented perspective.
  arXiv  Detail & Related papers  (2026-06-15T14:09:37Z)
• SEMA: Simple yet Effective Learning for Multi-Turn Jailbreak Attacks [53.97948802255959]
  We propose a framework that trains a multi-turn attacker without relying on any existing strategies or external data.<n>Prefilling self-tuning enables usable rollouts by fine-tuning on non-refusal, well-structured, multi-turn adversarial prompts.<n>We anchor harmful intent in multi-turn jailbreaks via an intent-drift-aware reward that combines intent alignment, compliance risk, and level of detail.
  arXiv  Detail & Related papers  (2026-02-06T16:44:57Z)
• Metaphor-based Jailbreaking Attacks on Text-to-Image Models [41.420325236578755]
  textbfMJA is a textbfmetaphor-based textbfjailbreaking textbfattack method inspired by the Taboo game.<n>It efficiently attacks diverse defense mechanisms without prior knowledge of their type.
  arXiv  Detail & Related papers  (2025-12-06T12:38:00Z)
• Multi-Faceted Attack: Exposing Cross-Model Vulnerabilities in Defense-Equipped Vision-Language Models [54.61181161508336]
  We introduce Multi-Faceted Attack (MFA), a framework that exposes general safety vulnerabilities in leading defense-equipped Vision-Language Models (VLMs)<n>The core component of MFA is the Attention-Transfer Attack (ATA), which hides harmful instructions inside a meta task with competing objectives.<n>MFA achieves a 58.5% success rate and consistently outperforms existing methods.
  arXiv  Detail & Related papers  (2025-11-20T07:12:54Z)
• MTAttack: Multi-Target Backdoor Attacks against Large Vision-Language Models [52.37749859972453]
  We propose MTAttack, the first multi-target backdoor attack framework for enforcing accurate multiple trigger-target mappings in LVLMs.<n> Experiments on popular benchmarks demonstrate a high success rate of MTAttack for multi-target attacks.<n>Our attack exhibits strong generalizability across datasets and robustness against backdoor defense strategies.
  arXiv  Detail & Related papers  (2025-11-13T09:00:21Z)
• MetaCipher: A Time-Persistent and Universal Multi-Agent Framework for Cipher-Based Jailbreak Attacks for LLMs [14.530593083777502]
  We propose MetaCipher, a low-cost, multi-agent jailbreak framework.<n>Within as few as 10 queries, MetaCipher achieves state-of-the-art attack success rates on recent malicious prompt benchmarks.
  arXiv  Detail & Related papers  (2025-06-27T18:15:56Z)
• Lifelong Safety Alignment for Language Models [33.90238075760236]
  We propose a lifelong safety alignment framework for jailbreaking defenses.<n>A Meta-Attacker is trained to actively discover novel jailbreaking strategies, and a Defender is trained to resist them.<n>Our framework reduces the Meta-Attacker's success rate to just 7%, enabling safer and more reliable deployment of LLMs.
  arXiv  Detail & Related papers  (2025-05-26T17:40:40Z)
• ELBA-Bench: An Efficient Learning Backdoor Attacks Benchmark for Large Language Models [55.93380086403591]
  Generative large language models are vulnerable to backdoor attacks.<n>$textitELBA-Bench$ allows attackers to inject backdoor through parameter efficient fine-tuning.<n>$textitELBA-Bench$ provides over 1300 experiments.
  arXiv  Detail & Related papers  (2025-02-22T12:55:28Z)
• Siren: A Learning-Based Multi-Turn Attack Framework for Simulating Real-World Human Jailbreak Behaviors [12.550678408719756]
  We propose a learning-based multi-turn attack framework designed to simulate real-world human jailbreak behaviors.<n>Experiments demonstrate that Siren achieves an attack success rate (ASR) of 90% with LLaMA-3-8B as the attacker.<n>We hope Siren inspires the development of stronger defenses against advanced multi-turn jailbreak attacks.
  arXiv  Detail & Related papers  (2025-01-24T05:31:27Z)
• White-box Multimodal Jailbreaks Against Large Vision-Language Models [61.97578116584653]
  We propose a more comprehensive strategy that jointly attacks both text and image modalities to exploit a broader spectrum of vulnerability within Large Vision-Language Models. Our attack method begins by optimizing an adversarial image prefix from random noise to generate diverse harmful responses in the absence of text input. An adversarial text suffix is integrated and co-optimized with the adversarial image prefix to maximize the probability of eliciting affirmative responses to various harmful instructions.
  arXiv  Detail & Related papers  (2024-05-28T07:13:30Z)
• Robust Prompt Optimization for Defending Language Models Against Jailbreaking Attacks [17.22989422489567]
  Large language models (LLMs) are vulnerable to adversarial attacks or jailbreaking. We propose an optimization-based objective for defending LLMs against jailbreaking attacks and an algorithm to create robust system-level defenses. Our results show improved robustness to both jailbreaks seen during optimization and unknown jailbreaks, reducing the attack success rate (ASR) on GPT-4 to 6% and Llama-2 to 0% on JailbreakBench.
  arXiv  Detail & Related papers  (2024-01-30T18:56:08Z)
• SmoothLLM: Defending Large Language Models Against Jailbreaking Attacks [99.23352758320945]
  We propose SmoothLLM, the first algorithm designed to mitigate jailbreaking attacks on large language models (LLMs) Based on our finding that adversarially-generated prompts are brittle to character-level changes, our defense first randomly perturbs multiple copies of a given input prompt, and then aggregates the corresponding predictions to detect adversarial inputs.
  arXiv  Detail & Related papers  (2023-10-05T17:01:53Z)
• Composite Adversarial Attacks [57.293211764569996]
  Adversarial attack is a technique for deceiving Machine Learning (ML) models. In this paper, a new procedure called Composite Adrial Attack (CAA) is proposed for automatically searching the best combination of attack algorithms. CAA beats 10 top attackers on 11 diverse defenses with less elapsed time.
  arXiv  Detail & Related papers  (2020-12-10T03:21:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.