Why does weak-OOD help? A Further Step Towards Understanding Jailbreaking VLMs

• URL: http://arxiv.org/abs/2511.08367v1
• Date: Wed, 12 Nov 2025 01:55:52 GMT
• Title: Why does weak-OOD help? A Further Step Towards Understanding Jailbreaking VLMs
• Abstract summary: Large Vision-Language Models (VLMs) are susceptible to jailbreak attacks.<n>This paper advances the in-depth understanding of OOD-based VLM jailbreak methods.
• Score: 62.03547713936256
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large Vision-Language Models (VLMs) are susceptible to jailbreak attacks: researchers have developed a variety of attack strategies that can successfully bypass the safety mechanisms of VLMs. Among these approaches, jailbreak methods based on the Out-of-Distribution (OOD) strategy have garnered widespread attention due to their simplicity and effectiveness. This paper further advances the in-depth understanding of OOD-based VLM jailbreak methods. Experimental results demonstrate that jailbreak samples generated via mild OOD strategies exhibit superior performance in circumventing the safety constraints of VLMs--a phenomenon we define as ''weak-OOD''. To unravel the underlying causes of this phenomenon, this study takes SI-Attack, a typical OOD-based jailbreak method, as the research object. We attribute this phenomenon to a trade-off between two dominant factors: input intent perception and model refusal triggering. The inconsistency in how these two factors respond to OOD manipulations gives rise to this phenomenon. Furthermore, we provide a theoretical argument for the inevitability of such inconsistency from the perspective of discrepancies between model pre-training and alignment processes. Building on the above insights, we draw inspiration from optical character recognition (OCR) capability enhancement--a core task in the pre-training phase of mainstream VLMs. Leveraging this capability, we design a simple yet highly effective VLM jailbreak method, whose performance outperforms that of SOTA baselines.

Related papers

• TrailBlazer: History-Guided Reinforcement Learning for Black-Box LLM Jailbreaking [10.964111261280413]
  We propose a history-aware RL-based jailbreak framework that analyzes and reweights vulnerability signals from prior steps to guide future decisions.<n>Our method achieves state-of-the-art jailbreak performance while significantly improving query efficiency.
  arXiv  Detail & Related papers  (2026-02-06T07:11:10Z)
• A Simple and Efficient Jailbreak Method Exploiting LLMs' Helpfulness [32.47621091096285]
  Safety alignment aims to prevent Large Language Models (LLMs) from responding to harmful queries.<n>In this paper, we introduce HILL, a novel jailbreak approach that transforms imperative harmful requests into learning-style questions.<n> Experiments on the AdvBench dataset across a wide range of models demonstrate HILL's strong effectiveness, generalizability, and harmfulness.
  arXiv  Detail & Related papers  (2025-09-17T04:21:20Z)
• Unraveling LLM Jailbreaks Through Safety Knowledge Neurons [26.157477756143166]
  We present a novel neuron-level interpretability method that focuses on the role of safety-related knowledge neurons.<n>We show that adjusting the activation of safety-related neurons can effectively control the model's behavior with a mean ASR higher than 97%.<n>We propose SafeTuning, a fine-tuning strategy that reinforces safety-critical neurons to improve model robustness.
  arXiv  Detail & Related papers  (2025-09-01T17:17:06Z)
• Towards Effective MLLM Jailbreaking Through Balanced On-Topicness and OOD-Intensity [24.809329513705915]
  Multimodal large language models (MLLMs) are widely used in vision-language reasoning tasks.<n> MLLMs are vulnerable to adversarial prompts as safety mechanisms fail to prevent the generation of harmful outputs.<n>We introduce a four-axis evaluation framework that considers input on-topicness, input out-of-distribution (OOD) intensity, output harmfulness, and output refusal rate.
  arXiv  Detail & Related papers  (2025-08-11T18:57:55Z)
• ARMOR: Aligning Secure and Safe Large Language Models via Meticulous Reasoning [64.32925552574115]
  ARMOR is a large language model that analyzes jailbreak strategies and extracts the core intent.<n> ARMOR achieves state-of-the-art safety performance, with an average harmful rate of 0.002 and an attack success rate of 0.06 against advanced optimization-based jailbreaks.
  arXiv  Detail & Related papers  (2025-07-14T09:05:54Z)
• ICLShield: Exploring and Mitigating In-Context Learning Backdoor Attacks [61.06621533874629]
  In-context learning (ICL) has demonstrated remarkable success in large language models (LLMs)<n>In this paper, we propose, for the first time, the dual-learning hypothesis, which posits that LLMs simultaneously learn both the task-relevant latent concepts and backdoor latent concepts.<n>Motivated by these findings, we propose ICLShield, a defense mechanism that dynamically adjusts the concept preference ratio.
  arXiv  Detail & Related papers  (2025-07-02T03:09:20Z)
• Cannot See the Forest for the Trees: Invoking Heuristics and Biases to Elicit Irrational Choices of LLMs [83.11815479874447]
  We propose a novel jailbreak attack framework, inspired by cognitive decomposition and biases in human cognition.<n>We employ cognitive decomposition to reduce the complexity of malicious prompts and relevance bias to reorganize prompts.<n>We also introduce a ranking-based harmfulness evaluation metric that surpasses the traditional binary success-or-failure paradigm.
  arXiv  Detail & Related papers  (2025-05-03T05:28:11Z)
• Reasoning-to-Defend: Safety-Aware Reasoning Can Defend Large Language Models from Jailbreaking [54.10710423370126]
  We propose Reasoning-to-Defend (R2D), a training paradigm that integrates a safety-aware reasoning mechanism into Large Language Models' generation process.<n>CPO enhances the model's perception of the safety status of given dialogues.<n>Experiments demonstrate that R2D effectively mitigates various attacks and improves overall safety, while maintaining the original performances.
  arXiv  Detail & Related papers  (2025-02-18T15:48:46Z)
• You Can't Eat Your Cake and Have It Too: The Performance Degradation of LLMs with Jailbreak Defense [34.023473699165315]
  We study the utility degradation, safety elevation, and exaggerated-safety escalation of LLMs with jailbreak defense strategies.<n>We find that mainstream jailbreak defenses fail to ensure both safety and performance simultaneously.
  arXiv  Detail & Related papers  (2025-01-21T15:24:29Z)
• The VLLM Safety Paradox: Dual Ease in Jailbreak Attack and Defense [56.32083100401117]
  The vulnerability of Vision Large Language Models (VLLMs) to jailbreak attacks appears as no surprise.<n>Recent defense mechanisms against these attacks have reached near-saturation performance on benchmark evaluations.
  arXiv  Detail & Related papers  (2024-11-13T07:57:19Z)
• $\textit{MMJ-Bench}$: A Comprehensive Study on Jailbreak Attacks and Defenses for Multimodal Large Language Models [11.02754617539271]
  We introduce textitMMJ-Bench, a unified pipeline for evaluating jailbreak attacks and defense techniques for MLLMs. We assess the effectiveness of various attack methods against SoTA MLLMs and evaluate the impact of defense mechanisms on both defense effectiveness and model utility.
  arXiv  Detail & Related papers  (2024-08-16T00:18:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.