Jailbreaking Vision-Language Models Through the Visual Modality

• URL: http://arxiv.org/abs/2605.00583v1
• Date: Fri, 01 May 2026 11:43:21 GMT
• Title: Jailbreaking Vision-Language Models Through the Visual Modality
• Abstract summary: We introduce four jailbreak attacks exploiting the vision component of vision-language models.<n>Our attacks bypass safety alignment and expose a cross-modality alignment gap.<n>These findings highlight that robust VLM alignment requires treating vision as a first-class target for safety post-training.
• Score: 16.986459949629126
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The visual modality of vision-language models (VLMs) is an underexplored attack surface for bypassing safety alignment. We introduce four jailbreak attacks exploiting the vision component: (1) encoding harmful instructions as visual symbol sequences with a decoding legend, (2) replacing harmful objects with benign substitutes (e.g., bomb -> banana) then prompting for harmful actions using the substitute term, (3) replacing harmful text in images (e.g., on book covers) with benign words while visual context preserves the original meaning, and (4) visual analogy puzzles whose solution requires inferring a prohibited concept. Evaluating across six frontier VLMs, our visual attacks bypass safety alignment and expose a cross-modality alignment gap: text-based safety training does not automatically generalize to harmful intent conveyed visually. For example, our visual cipher achieves 40.9% attack success on Claude-Haiku-4.5 versus 10.7% for an equivalent textual cipher. To further our insight into the attack mechanism, we present preliminary interpretability and mitigation results. These findings highlight that robust VLM alignment requires treating vision as a first-class target for safety post-training.

Related papers

• Visual Self-Fulfilling Alignment: Shaping Safety-Oriented Personas via Threat-Related Images [15.316817046650208]
  Multimodal large language models (MLLMs) face safety misalignment, where visual inputs enable harmful outputs.<n>Inspired by the Self-Fulfilling mechanism underlying emergent misalignment, we propose Visual Self-Fulfilling Alignment (VSFA)<n> VSFA fine-tunes vision-language models (VLMs) on neutral VQA tasks constructed around threat-related images, without any safety labels.
  arXiv  Detail & Related papers  (2026-03-09T15:20:53Z)
• Invisible Safety Threat: Malicious Finetuning for LLM via Steganography [74.00809267925642]
  A compromised large language model can maintain a facade of proper safety alignment while covertly generating harmful content.<n>We demonstrate this invisible safety threat on GPT-4.1 despite the OpenAI finetuning API's safeguards.<n>We quantitatively evaluate our method on the AdvBench dataset, using Llama-Guard-3-8B for content safety classification.
  arXiv  Detail & Related papers  (2026-03-09T08:48:27Z)
• VII: Visual Instruction Injection for Jailbreaking Image-to-Video Generation Models [57.128876964730644]
  Image-to-Video (I2V) generation models, which condition video generation on reference images, have shown emerging visual instruction-following capability.<n>We propose Visual Instruction Injection (VII), a training-free and transferable jailbreaking framework that disguises the malicious intent of unsafe text prompts as benign visual instructions in the safe reference image.<n>VII achieves Attack Success Rates of up to 83.5% while reducing Refusal Rates to near zero, significantly outperforming existing baselines.
  arXiv  Detail & Related papers  (2026-02-24T15:20:01Z)
• Multi-Faceted Attack: Exposing Cross-Model Vulnerabilities in Defense-Equipped Vision-Language Models [54.61181161508336]
  We introduce Multi-Faceted Attack (MFA), a framework that exposes general safety vulnerabilities in leading defense-equipped Vision-Language Models (VLMs)<n>The core component of MFA is the Attention-Transfer Attack (ATA), which hides harmful instructions inside a meta task with competing objectives.<n>MFA achieves a 58.5% success rate and consistently outperforms existing methods.
  arXiv  Detail & Related papers  (2025-11-20T07:12:54Z)
• VEIL: Jailbreaking Text-to-Video Models via Visual Exploitation from Implicit Language [25.38940067963429]
  Prior attacks on text-to-video (T2V) models typically add adversarial perturbations to obviously unsafe prompts.<n>We show that benign-looking prompts containing rich, implicit cues can induce T2V models to generate semantically unsafe videos.<n>We propose VEIL, a jailbreak framework that leverages T2V models' cross-modal associative patterns via a modular prompt design.
  arXiv  Detail & Related papers  (2025-11-17T08:31:43Z)
• Visual Contextual Attack: Jailbreaking MLLMs with Image-Driven Context Injection [31.1604742796343]
  multimodal large language models (MLLMs) have demonstrated tremendous potential for real-world applications.<n>Security vulnerabilities exhibited by the visual modality pose significant challenges to deploying such models in open-world environments.<n>We propose the VisCo (Visual Contextual) Attack, where visual information serves as a necessary component in constructing a vision-centric jailbreak context.
  arXiv  Detail & Related papers  (2025-07-03T17:53:12Z)
• TrojVLM: Backdoor Attack Against Vision Language Models [50.87239635292717]
  This study introduces TrojVLM, the first exploration of backdoor attacks aimed at Vision Language Models (VLMs) TrojVLM inserts predetermined target text into output text when encountering poisoned images. A novel semantic preserving loss is proposed to ensure the semantic integrity of the original image content.
  arXiv  Detail & Related papers  (2024-09-28T04:37:09Z)
• Jailbreak Vision Language Models via Bi-Modal Adversarial Prompt [60.54666043358946]
  This paper introduces the Bi-Modal Adversarial Prompt Attack (BAP), which executes jailbreaks by optimizing textual and visual prompts cohesively. In particular, we utilize a large language model to analyze jailbreak failures and employ chain-of-thought reasoning to refine textual prompts.
  arXiv  Detail & Related papers  (2024-06-06T13:00:42Z)
• White-box Multimodal Jailbreaks Against Large Vision-Language Models [61.97578116584653]
  We propose a more comprehensive strategy that jointly attacks both text and image modalities to exploit a broader spectrum of vulnerability within Large Vision-Language Models. Our attack method begins by optimizing an adversarial image prefix from random noise to generate diverse harmful responses in the absence of text input. An adversarial text suffix is integrated and co-optimized with the adversarial image prefix to maximize the probability of eliciting affirmative responses to various harmful instructions.
  arXiv  Detail & Related papers  (2024-05-28T07:13:30Z)
• ImgTrojan: Jailbreaking Vision-Language Models with ONE Image [37.80216561793555]
  We propose a novel jailbreaking attack against vision language models (VLMs)<n>A scenario where our poisoned (image, text) data pairs are included in the training data is assumed.<n>By replacing the original textual captions with malicious jailbreak prompts, our method can perform jailbreak attacks with the poisoned images.
  arXiv  Detail & Related papers  (2024-03-05T12:21:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.