Misaligned Roles, Misplaced Images: Structural Input Perturbations Expose Multimodal Alignment Blind Spots

• URL: http://arxiv.org/abs/2504.03735v1
• Date: Tue, 01 Apr 2025 03:54:36 GMT
• Title: Misaligned Roles, Misplaced Images: Structural Input Perturbations Expose Multimodal Alignment Blind Spots
• Abstract summary: We introduce Role-Modality Attacks (RMA), a class of adversarial attacks that exploit role confusion between the user and assistant.<n>RMAs manipulate the input structure without altering the query itself.<n>We propose an adversarial training approach that makes the model robust against input prompt perturbations.
• Score: 6.302734977940601
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multimodal Language Models (MMLMs) typically undergo post-training alignment to prevent harmful content generation. However, these alignment stages focus primarily on the assistant role, leaving the user role unaligned, and stick to a fixed input prompt structure of special tokens, leaving the model vulnerable when inputs deviate from these expectations. We introduce Role-Modality Attacks (RMA), a novel class of adversarial attacks that exploit role confusion between the user and assistant and alter the position of the image token to elicit harmful outputs. Unlike existing attacks that modify query content, RMAs manipulate the input structure without altering the query itself. We systematically evaluate these attacks across multiple Vision Language Models (VLMs) on eight distinct settings, showing that they can be composed to create stronger adversarial prompts, as also evidenced by their increased projection in the negative refusal direction in the residual stream, a property observed in prior successful attacks. Finally, for mitigation, we propose an adversarial training approach that makes the model robust against input prompt perturbations. By training the model on a range of harmful and benign prompts all perturbed with different RMA settings, it loses its sensitivity to Role Confusion and Modality Manipulation attacks and is trained to only pay attention to the content of the query in the input prompt structure, effectively reducing Attack Success Rate (ASR) while preserving the model's general utility.

Related papers

• Automatic Red Teaming for Implicit Vulnerabilities of Text-to-Image Models [58.59008920408855]
  We propose Adversarial Probing for Implicit VulnErabilities (AdvPIE) to expose implicit vulnerabilities without requiring access to the parameters of target models.<n> AdvPIE adopts a policy agent to generate and refine implicit adversarial prompts based on the feedback from a judge agent.<n>Experiments on standard and safety-aligned T2I models show that AdvPIE1 effectively uncovers implicit vulnerabilities.
  arXiv  Detail & Related papers  (2026-09-05T13:45:09Z)
• Who Pays the Price? Stakeholder-Centric Prompt Injection Benchmarking for Real-world Web Agents [93.19140872946842]
  Web agents driven by large language models (LLMs) are increasingly deployed in real-world environments.<n>This makes them vulnerable to prompt-injection attacks, in which seemingly benign content embeds adversarial instructions that manipulate agent behaviour.<n>We introduce textbfsysname, a benchmark to systematically categorize and attribute harm in real-world web agent systems.
  arXiv  Detail & Related papers  (2026-06-11T14:12:43Z)
• Attention Hijacking: Response Manipulation Across Queries in Vision-Language Models [57.870323273127234]
  Motivated by the observation, we propose textbfAttention Hijacking, a novel adversarial attack that steers internal attention toward a persistent image-dominant pattern.<n>Our method reduces the dependence of the manipulated output on the specific wording of the query.
  arXiv  Detail & Related papers  (2026-05-17T08:02:27Z)
• Visual Backdoor Attacks on MLLM Embodied Decision Making via Contrastive Trigger Learning [89.1856483797116]
  We introduce BEAT, the first framework to inject visual backdoors into MLLM-based embodied agents.<n>Unlike textual triggers, object triggers exhibit wide variation across viewpoints and lighting, making them difficult to implant reliably.<n>BEAT achieves attack success rates up to 80%, while maintaining strong benign task performance.
  arXiv  Detail & Related papers  (2025-10-31T16:50:49Z)
• TokenSwap: Backdoor Attack on the Compositional Understanding of Large Vision-Language Models [57.32952956674526]
  We introduce TokenSwap, a more evasive and stealthy backdoor attack on large vision-language models (LVLMs)<n>Instead of enforcing a fixed targeted content, TokenSwap subtly disrupts the understanding of object relationships in text.<n> TokenSwap achieves high attack success rates while maintaining superior evasiveness and stealthiness.
  arXiv  Detail & Related papers  (2025-09-29T10:19:22Z)
• Proactive Disentangled Modeling of Trigger-Object Pairings for Backdoor Defense [0.0]
  Deep neural networks (DNNs) and generative AI (GenAI) are increasingly vulnerable to backdoor attacks.<n>In this paper, we introduce DBOM, a proactive framework that leverages structured disentanglement to identify and neutralize both seen and unseen backdoor threats.<n>We show that DBOM robustly detects poisoned images prior to downstream training, significantly enhancing the security of training pipelines.
  arXiv  Detail & Related papers  (2025-08-03T21:58:15Z)
• Robust Anti-Backdoor Instruction Tuning in LVLMs [53.766434746801366]
  We introduce a lightweight, certified-agnostic defense framework for large visual language models (LVLMs)<n>Our framework finetunes only adapter modules and text embedding layers under instruction tuning.<n>Experiments against seven attacks on Flickr30k and MSCOCO demonstrate that ours reduces their attack success rate to nearly zero.
  arXiv  Detail & Related papers  (2025-06-04T01:23:35Z)
• Separator Injection Attack: Uncovering Dialogue Biases in Large Language Models Caused by Role Separators [18.00739443633401]
  Role separators are employed to distinguish between different participants in a conversation. Misusing roles can lead to prompt injection attacks, which can easily misalign the model's behavior with the user's intentions. We develop the Separators Injection Attack (SIA), a new orthometric attack based on role separators.
  arXiv  Detail & Related papers  (2025-04-08T05:20:56Z)
• MIRAGE: Multimodal Immersive Reasoning and Guided Exploration for Red-Team Jailbreak Attacks [85.3303135160762]
  MIRAGE is a novel framework that exploits narrative-driven context and role immersion to circumvent safety mechanisms in Multimodal Large Language Models.<n>It achieves state-of-the-art performance, improving attack success rates by up to 17.5% over the best baselines.<n>We demonstrate that role immersion and structured semantic reconstruction can activate inherent model biases, facilitating the model's spontaneous violation of ethical safeguards.
  arXiv  Detail & Related papers  (2025-03-24T20:38:42Z)
• Prompt-driven Transferable Adversarial Attack on Person Re-Identification with Attribute-aware Textual Inversion [17.18411620606476]
  We introduce the Attribute-aware Prompt Attack (AP-Attack) to disrupt fine-grained semantic features of pedestrian images.<n>AP-Attack achieves state-of-the-art transferability, significantly outperforming previous methods by 22.9% on mean Drop Rate.
  arXiv  Detail & Related papers  (2025-02-27T02:32:58Z)
• Reformulation is All You Need: Addressing Malicious Text Features in DNNs [53.45564571192014]
  We propose a unified and adaptive defense framework that is effective against both adversarial and backdoor attacks.<n>Our framework outperforms existing sample-oriented defense baselines across a diverse range of malicious textual features.
  arXiv  Detail & Related papers  (2025-02-02T03:39:43Z)
• Extend Adversarial Policy Against Neural Machine Translation via Unknown Token [66.40609413186122]
  We propose the DexChar policy' that introduces character perturbations for the existing mainstream adversarial policy based on token substitution. We also improve the self-supervised matching that provides feedback in RL to cater to the semantic constraints required during training adversaries.
  arXiv  Detail & Related papers  (2025-01-21T14:43:04Z)
• Disrupting Diffusion: Token-Level Attention Erasure Attack against Diffusion-based Customization [19.635385099376066]
  malicious users have misused diffusion-based customization methods like DreamBooth to create fake images. In this paper, we propose DisDiff, a novel adversarial attack method to disrupt the diffusion model outputs.
  arXiv  Detail & Related papers  (2024-05-31T02:45:31Z)
• Learning diverse attacks on large language models for robust red-teaming and safety tuning [126.32539952157083]
  Red-teaming, or identifying prompts that elicit harmful responses, is a critical step in ensuring the safe deployment of large language models.<n>We show that even with explicit regularization to favor novelty and diversity, existing approaches suffer from mode collapse or fail to generate effective attacks.<n>We propose to use GFlowNet fine-tuning, followed by a secondary smoothing phase, to train the attacker model to generate diverse and effective attack prompts.
  arXiv  Detail & Related papers  (2024-05-28T19:16:17Z)
• VL-Trojan: Multimodal Instruction Backdoor Attacks against Autoregressive Visual Language Models [65.23688155159398]
  Autoregressive Visual Language Models (VLMs) showcase impressive few-shot learning capabilities in a multimodal context. Recently, multimodal instruction tuning has been proposed to further enhance instruction-following abilities. Adversaries can implant a backdoor by injecting poisoned samples with triggers embedded in instructions or images. We propose a multimodal instruction backdoor attack, namely VL-Trojan.
  arXiv  Detail & Related papers  (2024-02-21T14:54:30Z)
• ExploreADV: Towards exploratory attack for Neural Networks [0.33302293148249124]
  ExploreADV is a general and flexible adversarial attack system that is capable of modeling regional and imperceptible attacks. We show that our system offers users good flexibility to focus on sub-regions of inputs, explore imperceptible perturbations and understand the vulnerability of pixels/regions to adversarial attacks.
  arXiv  Detail & Related papers  (2023-01-01T07:17:03Z)
• Adversarial Visual Robustness by Causal Intervention [56.766342028800445]
  Adversarial training is the de facto most promising defense against adversarial examples. Yet, its passive nature inevitably prevents it from being immune to unknown attackers. We provide a causal viewpoint of adversarial vulnerability: the cause is the confounder ubiquitously existing in learning.
  arXiv  Detail & Related papers  (2021-06-17T14:23:54Z)
• A Self-supervised Approach for Adversarial Robustness [105.88250594033053]
  Adversarial examples can cause catastrophic mistakes in Deep Neural Network (DNNs) based vision systems. This paper proposes a self-supervised adversarial training mechanism in the input space. It provides significant robustness against the textbfunseen adversarial attacks.
  arXiv  Detail & Related papers  (2020-06-08T20:42:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.