Do VLMs Share Safety Neurons Across Modalities?

• URL: http://arxiv.org/abs/2608.30750v1
• Date: Mon, 31 Aug 2026 13:17:41 GMT
• Title: Do VLMs Share Safety Neurons Across Modalities?
• Abstract summary: Vision-language models (VLMs) can comply with harmful requests delivered through images, even when their backbone LLMs would refuse the same content in text.<n>We propose a two-stage detection pipeline with iterative ablation that accounts for self-repair.<n>We introduce two modality-isolated benchmarks, ViSafe-Detect and ViSafe-Eval, which decouple visual and textual safety signals.
• Score: 13.44337685375747
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision-language models (VLMs) can comply with harmful requests delivered through images, even when their LLM backbones would refuse the same content in text. While prior work characterizes these jailbreaks empirically or at the representation level, how visual inputs perturb safety pathways at the neuron level remains uncharted. We close this gap with a causal, neuron-level analysis of safety mechanisms in 10 VLMs. We propose a two-stage detection pipeline with iterative ablation that accounts for self-repair, and introduce two modality-isolated benchmarks, ViSafe-Detect and ViSafe-Eval, which decouple visual and textual safety signals. Our analysis reveals: (i) Text safety in VLMs is localizable: $\sim$88 neurons ($<$0.01%) whose targeted ablation substantially reduces refusal. (ii) Text safety neurons constitute the dominant refusal pathway: ablating them is the only intervention that consistently and substantially reduces refusal across all models. (iii) Visual safety is high-dimensional and diffuse at the single-neuron level: text safety concentrates in $\sim$5 subspace directions while visual safety requires $\geq$50. This gap holds across architectures, explaining why current alignment has not closed the visual safety gap. Project page is at: https://jiaxuan-li.github.io/vlm-safety-neuron/ Warning: this paper may include examples of harmful content.

Related papers

• No Single Neuron of Failure: Distributed Safety Alignment Against White-Box Attacks [61.07406641465909]
  We propose distributed safety alignment, which encodes safety capabilities across multiple computational neurons.<n>We show that DSA substantially improves against white-box neuron-level safety attacks while preserving the model's general language and multimodal utility.
  arXiv  Detail & Related papers  (2026-08-02T17:49:07Z)
• One Anchor for All: Unified Multilingual and Multimodal Safety Alignment for LVLMs [63.29484454957031]
  We propose a neuron-level cross-dimensional safety alignment framework driven by modality- and language-shared safety neurons (MLS-Neurons)<n>Our method significantly outperforms state-of-the-art approaches across diverse multilingual and multimodal safety benchmarks.
  arXiv  Detail & Related papers  (2026-07-30T09:30:03Z)
• AEGIS: A Mechanism-Guided Defense against Visual Synonym Jailbreaks in Text-to-Image Models [22.925421768193956]
  AEGIS is an inference-time defense that applies similarity-aware repulsion only at the identified vulnerable heads.<n>It preserves benign fidelity, avoids suppressing hard-negative concepts, and transfers to SD 2.1 and FLUX.1 after re-identifying the critical heads for each backbone.
  arXiv  Detail & Related papers  (2026-07-07T10:27:54Z)
• Precise Shield: Explaining and Aligning VLLM Safety via Neuron-Level Guidance [64.18179414483966]
  In real-world deployments, Vision-Language Large Models (VLLMs) face critical challenges from multilingual and multimodal composite attacks.<n>We propose Precise Shield, a two-stage framework that first identifies safety neurons by contrasting activation patterns between harmful and benign inputs.<n>This strategy substantially improves safety while preserving multilingual and multimodal generalization.
  arXiv  Detail & Related papers  (2026-04-10T02:42:52Z)
• Visual Self-Fulfilling Alignment: Shaping Safety-Oriented Personas via Threat-Related Images [15.316817046650208]
  Multimodal large language models (MLLMs) face safety misalignment, where visual inputs enable harmful outputs.<n>Inspired by the Self-Fulfilling mechanism underlying emergent misalignment, we propose Visual Self-Fulfilling Alignment (VSFA)<n> VSFA fine-tunes vision-language models (VLMs) on neutral VQA tasks constructed around threat-related images, without any safety labels.
  arXiv  Detail & Related papers  (2026-03-09T15:20:53Z)
• HoliSafe: Holistic Safety Benchmarking and Modeling for Vision-Language Model [58.12612140992874]
  We introduce a holistic safety dataset and benchmark, textbfHoliSafe, that spans all five safe/unsafe image-text combinations.<n>We also propose a novel modular framework for enhancing VLM safety with a visual guard module (VGM) designed to assess the harmfulness of input images.<n> Experiments show that Safe-VLM with VGM, trained on our HoliSafe, achieves state-of-the-art safety performance across multiple VLM benchmarks.
  arXiv  Detail & Related papers  (2025-06-05T07:26:34Z)
• Safe + Safe = Unsafe? Exploring How Safe Images Can Be Exploited to Jailbreak Large Vision-Language Models [80.77246856082742]
  Safety Snowball Agent (SSA) is a novel agent-based framework leveraging agents' autonomous and tool-using abilities to jailbreak LVLMs.<n>Our experiments demonstrate that ours can use nearly any image to induce LVLMs to produce unsafe content, achieving high success jailbreaking rates against the latest LVLMs.
  arXiv  Detail & Related papers  (2024-11-18T11:58:07Z)
• Towards Understanding Safety Alignment: A Mechanistic Perspective from Safety Neurons [57.07507194465299]
  Large language models (LLMs) excel in various capabilities but pose safety risks such as generating harmful content and misinformation, even after safety alignment.<n>We focus on identifying and analyzing safety neurons within LLMs that are responsible for safety behaviors.<n>We propose inference-time activation contrasting to locate these neurons and dynamic activation patching to evaluate their causal effects on model safety.
  arXiv  Detail & Related papers  (2024-06-20T09:35:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.