One Anchor for All: Unified Multilingual and Multimodal Safety Alignment for LVLMs

• URL: http://arxiv.org/abs/2607.27917v1
• Date: Thu, 30 Jul 2026 09:30:03 GMT
• Title: One Anchor for All: Unified Multilingual and Multimodal Safety Alignment for LVLMs
• Abstract summary: We propose a neuron-level cross-dimensional safety alignment framework driven by modality- and language-shared safety neurons (MLS-Neurons)<n>Our method significantly outperforms state-of-the-art approaches across diverse multilingual and multimodal safety benchmarks.
• Score: 63.29484454957031
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: As large vision-language models (LVLMs) are deployed globally, the combination of multilingual instructions and visual information makes malicious attacks more covert and sophisticated than ever before. However, existing methods isolate language and modality defenses, which, coupled with the scarcity of safety data and high fine-tuning costs, makes it difficult for models to defend against compound attacks. To address this severe challenge, we propose a neuron-level cross-dimensional safety alignment framework driven by modality- and language-shared safety neurons (MLS-Neurons). First, we identify monolingual and unimodal safety neurons by comparing responses to harmful and benign samples, quantifying functional saliency through activation strength and downstream impact. Then, by intersecting these unimodal neurons within each language, we extract modality-shared safety neurons (MS-Neurons) responsive to both visual and textual risks, bridging the safety representation gap between modalities. Furthermore, using English as a semantic anchor, we intersect MS-Neurons across languages to identify modality- and language-shared safety neurons (MLS-Neurons), serving as key defenses against compound attacks. Finally, we update only this minimal subset of shared neurons (~0.03% of parameters), transferring English-only safety supervision to multilingual and multimodal scenarios. Extensive experiments show that our method significantly outperforms state-of-the-art approaches across diverse multilingual and multimodal safety benchmarks while preserving general utility.

Related papers

• The Illusion of Cross-Lingual Safety in Low-Resource Languages [7.471112976694495]
  Safety alignment in large language models (LLMs) is largely developed in English, assuming these safeguards generalize across multilingual settings.<n>We investigate cross-lingual safety transfer in four African languages, Twi, Hausa, Amharic, and Swahili, using LoDNA, a new safety dataset that pairs literal translations with culturally localized prompts.<n>Our experimental results show that cross-lingual safety transfer is severely limited; harmful prompts retain less than 10% of the English refusal signal across most language-model pairs.
  arXiv  Detail & Related papers  (2026-08-11T17:05:26Z)
• No Single Neuron of Failure: Distributed Safety Alignment Against White-Box Attacks [61.07406641465909]
  We propose distributed safety alignment, which encodes safety capabilities across multiple computational neurons.<n>We show that DSA substantially improves against white-box neuron-level safety attacks while preserving the model's general language and multimodal utility.
  arXiv  Detail & Related papers  (2026-08-02T17:49:07Z)
• SafeNexus: Discovering and Steering Modality-Universal Safety Neurons in MLLMs [69.51621286001262]
  We introduce SafeNexus, a cross-modal safety alignment framework that adopts a dedicated neuron-level intervention strategy.<n>We identify modality-bound safety neurons (BS-Neurons), and validate their role in regulating safety behavior within each modality.<n>We propose two safety alignment strategies: activation-level safety amplifier and safety neuron calibrator.
  arXiv  Detail & Related papers  (2026-07-31T02:45:56Z)
• Exploring Adversarial Robustness and Safety Alignment in Multilingual Multi-Modal Large Language Models [37.24612885977265]
  Multimodal Large Language Models integrate visual perception into language reasoning, introducing a continuous attack surface susceptible to adversarial attacks.<n>Prior work on MLLM has focused largely on English-centric tasks, leaving multilingual behaviour unexplored.<n>We study adversarial robustness and multimodal safety across 12 diverse languages, evaluating open-source MLLMs that acquire multilingual capability through instruction tuning.
  arXiv  Detail & Related papers  (2026-06-02T15:42:10Z)
• Multilingual Safety Alignment via Self-Distillation [17.94152626632751]
  Large language models (LLMs) exhibit severe multilingual safety misalignment.<n>We propose a cross-lingual safeguard transfer framework named Multilingual Self-Distillation (MSD)<n>Our framework is flexible and can be integrated with different self-distillation strategies.
  arXiv  Detail & Related papers  (2026-05-03T14:22:49Z)
• Multilingual Refusal Alignment for Safer Large Language Models [53.64286756804503]
  We investigate whether single-language alignment transfers cross-lingually, how language consistency is preserved during training, and the resulting trade-offs with general knowledge capabilities.<n>We introduce RefusEU, a novel refusal alignment dataset covering 12 European languages, including a dedicated test set for evaluating current state-of-the-art models.<n>Our controlled Direct Preference Optimization (DPO) experiments provide two key insights: aligning models exclusively in English is insufficient to ensure cross-lingual safety, even for the same harm categories, whereas training on multilingual datasets can improve safety without degrading general performance, as measured by the Global MMLU benchmark.
  arXiv  Detail & Related papers  (2026-04-24T09:29:14Z)
• Precise Shield: Explaining and Aligning VLLM Safety via Neuron-Level Guidance [64.18179414483966]
  In real-world deployments, Vision-Language Large Models (VLLMs) face critical challenges from multilingual and multimodal composite attacks.<n>We propose Precise Shield, a two-stage framework that first identifies safety neurons by contrasting activation patterns between harmful and benign inputs.<n>This strategy substantially improves safety while preserving multilingual and multimodal generalization.
  arXiv  Detail & Related papers  (2026-04-10T02:42:52Z)
• Who Transfers Safety? Identifying and Targeting Cross-Lingual Shared Safety Neurons [49.772147495578736]
  Cross-lingual shared safety neurons (SS-Neurons) regulate safety behavior across languages.<n>We propose a neuron-oriented training strategy that targets SS-Neurons based on language resource distribution and model architecture.
  arXiv  Detail & Related papers  (2026-02-01T15:28:02Z)
• Align is not Enough: Multimodal Universal Jailbreak Attack against Multimodal Large Language Models [83.80177564873094]
  We propose a unified multimodal universal jailbreak attack framework.<n>We evaluate the undesirable context generation of MLLMs like LLaVA, Yi-VL, MiniGPT4, MiniGPT-v2, and InstructBLIP.<n>This study underscores the urgent need for robust safety measures in MLLMs.
  arXiv  Detail & Related papers  (2025-06-02T04:33:56Z)
• MPO: Multilingual Safety Alignment via Reward Gap Optimization [88.76638442683391]
  Large language models (LLMs) have become increasingly central to AI applications worldwide.<n>Existing preference learning methods for safety alignment, such as RLHF and DPO, are primarily monolingual and struggle with noisy multilingual data.<n>We introduce Multilingual reward gaP Optimization (MPO), a novel approach that leverages the well-aligned safety capabilities of the dominant language (English) to improve safety alignment across multiple languages.
  arXiv  Detail & Related papers  (2025-05-22T16:24:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.