Comparative Analysis of Patch Attack on VLM-Based Autonomous Driving Architectures

• URL: http://arxiv.org/abs/2603.08897v1
• Date: Mon, 09 Mar 2026 20:04:13 GMT
• Title: Comparative Analysis of Patch Attack on VLM-Based Autonomous Driving Architectures
• Abstract summary: This paper presents a systematic framework for comparative adversarial evaluation across three VLM architectures: Dolphins, OmniDrive ( Omni-L), and LeapVAD.<n>Using black-box optimization with semantic homogenization for fair comparison, we evaluate physically realizable patch attacks in CARLA simulation.<n>Results reveal severe vulnerabilities across all architectures, sustained multi-frame failures, and critical object detection degradation.
• Score: 8.340004072808005
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision-language models are emerging for autonomous driving, yet their robustness to physical adversarial attacks remains unexplored. This paper presents a systematic framework for comparative adversarial evaluation across three VLM architectures: Dolphins, OmniDrive (Omni-L), and LeapVAD. Using black-box optimization with semantic homogenization for fair comparison, we evaluate physically realizable patch attacks in CARLA simulation. Results reveal severe vulnerabilities across all architectures, sustained multi-frame failures, and critical object detection degradation. Our analysis exposes distinct architectural vulnerability patterns, demonstrating that current VLM designs inadequately address adversarial threats in safety-critical autonomous driving applications.

Related papers

• Breaking the weakest link to evade vision language models [0.254890465057467]
  Vision Language Models (VLMs) have emerged as a critical component of multimodal AI systems.<n>In this work, we investigate the vulnerability of VLMs to adversarial perturbations applied to visual inputs.<n>We show that small, human-imperceptible perturbations can substantially alter the textual interpretation produced by the models.
  arXiv  Detail & Related papers  (2026-08-19T14:06:31Z)
• Understanding Adversarial Transferability in Vision-Language Models for Autonomous Driving: A Cross-Architecture Analysis [1.3980986259786221]
  Vision-language models (VLMs) are increasingly used in autonomous driving.<n>They combine visual perception with language-based reasoning, supporting more interpretable decision-making.<n> adversarial transferability in VLM-based driving is studied.
  arXiv  Detail & Related papers  (2026-04-30T04:33:38Z)
• Semantic Misalignment in Vision-Language Models under Perceptual Degradation [2.9140696506330723]
  We study semantic misalignment in Vision-Language Models (VLMs) under controlled degradation of visual perception.<n>We observe severe failures in downstream VLM behavior, including hallucinated object mentions, omission of safety-critical entities, and inconsistent safety judgments.<n>Our results reveal a clear disconnect between pixel-level robustness and multimodal semantic reliability, highlighting a critical limitation of current VLM-based systems.
  arXiv  Detail & Related papers  (2026-01-13T09:13:05Z)
• AttackVLA: Benchmarking Adversarial and Backdoor Attacks on Vision-Language-Action Models [60.39655329875822]
  Vision-Language-Action (VLA) models enable robots to interpret natural-language instructions and perform diverse tasks.<n>Despite growing interest in attacking such models, the effectiveness of existing techniques remains unclear.<n>We propose AttackVLA, a unified framework that aligns with the VLA development lifecycle.
  arXiv  Detail & Related papers  (2025-11-15T10:30:46Z)
• Universal Camouflage Attack on Vision-Language Models for Autonomous Driving [67.34987318443761]
  Visual language modeling for automated driving is emerging as a promising research direction.<n>VLM-AD remains vulnerable to serious security threats from adversarial attacks.<n>We propose the first Universal Camouflage Attack framework for VLM-AD.
  arXiv  Detail & Related papers  (2025-09-24T14:52:01Z)
• FreezeVLA: Action-Freezing Attacks against Vision-Language-Action Models [124.02734355214325]
  Vision-Language-Action (VLA) models are driving rapid progress in robotics.<n> adversarial images can "freeze" VLA models and cause them to ignore subsequent instructions.<n>FreezeVLA generates and evaluates action-freezing attacks via min-max bi-level optimization.
  arXiv  Detail & Related papers  (2025-09-24T08:15:28Z)
• Integrated Simulation Framework for Adversarial Attacks on Autonomous Vehicles [42.02003282828958]
  This paper introduces a novel, open-source integrated simulation framework designed to generate adversarial attacks targeting both perception and communication layers of AVs.<n>Our implementation supports diverse perception-level attacks on LiDAR sensor data, along with communication-level threats such as V2X message manipulation and GPS spoofing.<n>We demonstrate the framework's effectiveness by evaluating the impact of generated adversarial scenarios on a state-of-the-art 3D object detector.
  arXiv  Detail & Related papers  (2025-08-31T20:53:08Z)
• Natural Reflection Backdoor Attack on Vision Language Model for Autonomous Driving [55.96227460521096]
  Vision-Language Models (VLMs) have been integrated into autonomous driving systems to enhance reasoning capabilities.<n>We propose a natural reflection-based backdoor attack targeting VLM systems in autonomous driving scenarios.<n>Our findings uncover a new class of attacks that exploit the stringent real-time requirements of autonomous driving.
  arXiv  Detail & Related papers  (2025-05-09T20:28:17Z)
• Black-Box Adversarial Attack on Vision Language Models for Autonomous Driving [65.61999354218628]
  We take the first step toward designing black-box adversarial attacks specifically targeting vision-language models (VLMs) in autonomous driving systems.<n>We propose Cascading Adversarial Disruption (CAD), which targets low-level reasoning breakdown by generating and injecting semantics.<n>We present Risky Scene Induction, which addresses dynamic adaptation by leveraging a surrogate VLM to understand and construct high-level risky scenarios.
  arXiv  Detail & Related papers  (2025-01-23T11:10:02Z)
• MirrorCheck: Efficient Adversarial Defense for Vision-Language Models [55.73581212134293]
  We propose a novel, yet elegantly simple approach for detecting adversarial samples in Vision-Language Models. Our method leverages Text-to-Image (T2I) models to generate images based on captions produced by target VLMs. Empirical evaluations conducted on different datasets validate the efficacy of our approach.
  arXiv  Detail & Related papers  (2024-06-13T15:55:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.