Challenging Vision-Language Models with Physically Deployable Multimodal Semantic Lighting Attacks

• URL: http://arxiv.org/abs/2604.12833v1
• Date: Tue, 14 Apr 2026 14:52:15 GMT
• Title: Challenging Vision-Language Models with Physically Deployable Multimodal Semantic Lighting Attacks
• Abstract summary: Vision-Language Models (VLMs) have shown remarkable performance, yet their security remains insufficiently understood.<n>Existing adversarial studies focus almost exclusively on the digital setting, leaving physical-world threats largely unexplored.<n>We propose Multimodal Semantic Lighting Attacks (MSLA), the first physically deployable adversarial attack framework against VLMs.
• Score: 23.938024446316717
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision-Language Models (VLMs) have shown remarkable performance, yet their security remains insufficiently understood. Existing adversarial studies focus almost exclusively on the digital setting, leaving physical-world threats largely unexplored. As VLMs are increasingly deployed in real environments, this gap becomes critical, since adversarial perturbations must be physically realizable. Despite this practical relevance, physical attacks against VLMs have not been systematically studied. Such attacks may induce recognition failures and further disrupt multimodal reasoning, leading to severe semantic misinterpretation in downstream tasks. Therefore, investigating physical attacks on VLMs is essential for assessing their real-world security risks. To address this gap, we propose Multimodal Semantic Lighting Attacks (MSLA), the first physically deployable adversarial attack framework against VLMs. MSLA uses controllable adversarial lighting to disrupt multimodal semantic understanding in real scenes, attacking semantic alignment rather than only task-specific outputs. Consequently, it degrades zero-shot classification performance of mainstream CLIP variants while inducing severe semantic hallucinations in advanced VLMs such as LLaVA and BLIP across image captioning and visual question answering (VQA). Extensive experiments in both digital and physical domains demonstrate that MSLA is effective, transferable, and practically realizable. Our findings provide the first evidence that VLMs are highly vulnerable to physically deployable semantic attacks, exposing a previously overlooked robustness gap and underscoring the urgent need for physical-world robustness evaluation of VLMs.

Related papers

• Forbid Your Attention: Fooling Multimodal Large Language Models by Selectively Removing Intrinsic Focus in Spectral Domain [94.64605746540472]
  Multimodal large language models (MLLMs) have extended the capability of large language models (LLMs) to process more contextual multimodal information.<n>Recent studies reveal that MLLMs remain highly vulnerable to adversarial inputs, especially those targeting visual components.<n>We propose a novel phase-aware adversarial attack framework that explicitly restricts adversarial perturbations to structure-relevant phase regions.
  arXiv  Detail & Related papers  (2026-09-01T06:34:32Z)
• Breaking the weakest link to evade vision language models [0.254890465057467]
  Vision Language Models (VLMs) have emerged as a critical component of multimodal AI systems.<n>In this work, we investigate the vulnerability of VLMs to adversarial perturbations applied to visual inputs.<n>We show that small, human-imperceptible perturbations can substantially alter the textual interpretation produced by the models.
  arXiv  Detail & Related papers  (2026-08-19T14:06:31Z)
• Devil in the Lens: Analyzing and Defending Physical Prompt Injection Against Vision-Language Models on Wearable Devices [7.183439319638428]
  Vision-Language Models (VLMs) are rapidly deployed on human-facing wearable devices such as smart glasses to enable multimodal perception and AI-assisted decision-making.<n>Our work characterizes how malicious textual information embedded in physical environments introduces a high-priority visual channel for prompt injection.<n>Using physically captured photos from AI glasses in over 200 real-world environments, our analysis identifies 6 representative threat vectors of physically injected prompts.<n>Results show that these attacks consistently manipulate model outputs across integrity- and safety-critical tasks, achieving attack success rates of up to 96% and 60% in simulated and real-world settings.
  arXiv  Detail & Related papers  (2026-07-11T11:56:05Z)
• Uncovering Linguistic Fragility in Vision-Language-Action Models via Diversity-Aware Red Teaming [64.48633529149579]
  We propose a novel framework to expose the vulnerabilities of Vision-Language-Action (VLA) models against linguistic variations.<n>Our method reduces the average task success rate from 93.33% to 5.85%, demonstrating a scalable approach to stress-testing VLA agents.
  arXiv  Detail & Related papers  (2026-04-07T08:43:36Z)
• Understanding and Enhancing Encoder-based Adversarial Transferability against Large Vision-Language Models [19.899086203883254]
  Large vision-language models (LVLMs) have achieved impressive success across multimodal tasks.<n>Their reliance on visual inputs exposes them to significant adversarial threats.<n>Existing encoder-based attacks perturb the input image by optimizing solely on the vision encoder, rather than the entire LVLM.<n>We present the first systematic study towards encoder-based adversarial transferability in LVLMs.
  arXiv  Detail & Related papers  (2026-02-10T05:51:02Z)
• Universal Camouflage Attack on Vision-Language Models for Autonomous Driving [67.34987318443761]
  Visual language modeling for automated driving is emerging as a promising research direction.<n>VLM-AD remains vulnerable to serious security threats from adversarial attacks.<n>We propose the first Universal Camouflage Attack framework for VLM-AD.
  arXiv  Detail & Related papers  (2025-09-24T14:52:01Z)
• ADVEDM:Fine-grained Adversarial Attack against VLM-based Embodied Agents [40.066839771776046]
  Vision-Language Models (VLMs) are widely used in embodied decision-making tasks.<n>Recent research has explored adversarial attacks on VLMs to reveal their vulnerabilities.<n>We propose a fine-grained adversarial attack framework, ADVEDM, which modifies the VLM's perception of only a few key objects.
  arXiv  Detail & Related papers  (2025-09-20T11:48:11Z)
• Transferable Adversarial Attacks on Black-Box Vision-Language Models [63.22532779621001]
  adversarial attacks can transfer from open-source to proprietary black-box models in text-only and vision-only contexts.<n>We show that attackers can craft perturbations to induce specific attacker-chosen interpretations of visual information.<n>We discover that universal perturbations -- modifications applicable to a wide set of images -- can consistently induce these misinterpretations.
  arXiv  Detail & Related papers  (2025-05-02T06:51:11Z)
• Manipulation Facing Threats: Evaluating Physical Vulnerabilities in End-to-End Vision Language Action Models [79.60405008196139]
  Vision Language Action Models (VLAMs) are being proposed to achieve better performance in open-vocabulary scenarios for robotic manipulation tasks.<n>In this paper, we comprehensively evaluate VLAMs in the face of potential physical threats.
  arXiv  Detail & Related papers  (2024-09-20T03:02:05Z)
• Visual Adversarial Examples Jailbreak Aligned Large Language Models [66.53468356460365]
  We show that the continuous and high-dimensional nature of the visual input makes it a weak link against adversarial attacks. We exploit visual adversarial examples to circumvent the safety guardrail of aligned LLMs with integrated vision. Our study underscores the escalating adversarial risks associated with the pursuit of multimodality.
  arXiv  Detail & Related papers  (2023-06-22T22:13:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.