Understanding Adversarial Transferability in Vision-Language Models for Autonomous Driving: A Cross-Architecture Analysis

• URL: http://arxiv.org/abs/2604.27414v1
• Date: Thu, 30 Apr 2026 04:33:38 GMT
• Title: Understanding Adversarial Transferability in Vision-Language Models for Autonomous Driving: A Cross-Architecture Analysis
• Abstract summary: Vision-language models (VLMs) are increasingly used in autonomous driving.<n>They combine visual perception with language-based reasoning, supporting more interpretable decision-making.<n> adversarial transferability in VLM-based driving is studied.
• Score: 1.3980986259786221
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision-language models (VLMs) are increasingly used in autonomous driving because they combine visual perception with language-based reasoning, supporting more interpretable decision-making, yet their robustness to physical adversarial attacks, especially whether such attacks transfer across different VLM architectures, is not well understood and poses a practical risk when attackers do not know which model a vehicle uses. We address this gap with a systematic cross-architecture study of adversarial transferability in VLM-based driving, evaluating three representative architectures (Dolphins, OmniDrive, and LeapVAD) using physically realizable patches placed on roadside infrastructure in both crosswalk and highway scenarios. Our transfer-matrix evaluation shows high cross-architecture effectiveness, with transfer rates of 73-91% (mean TR = 0.815 for crosswalk and 0.833 for highway) and sustained frame-level manipulation over 64.7-79.4% of the critical decision window even when patches are not optimized for the target model.

Related papers

• Development of Vision-Language Model-based GNSS Spoofing Detection for Autonomous Vehicle Navigation [10.15536470870511]
  We develop the first Vision-Language Model (VLM)-based framework for spoofing detection for autonomous vehicles.<n>Our approach introduces a three-stage fine-tuning process that first grounds visual cues, and then calibrates sensor data.<n>Results show that our VLM-based approach correctly classified every wrong-turn and stop attacks, and attains 88%-93% accuracy for overshoot attacks.
  arXiv  Detail & Related papers  (2026-07-27T03:20:08Z)
• AD4AD: Benchmarking Visual Anomaly Detection Models for Safer Autonomous Driving [10.121649137872298]
  Failures in autonomous driving translate directly into physical risk for passengers, pedestrians, and other road users.<n>Visual Anomaly Detection (VAD) enables the identification of anomalous objects not present during training.<n>VAD models produce pixel-level anomaly maps that can guide driver attention to specific regions of concern.
  arXiv  Detail & Related papers  (2026-04-16T17:54:53Z)
• UniDriveVLA: Unifying Understanding, Perception, and Action Planning for Autonomous Driving [48.461267171124945]
  We propose UniDriveVLA, a Unified Driving Vision-Language-Action model for autonomous driving.<n>It comprises three experts for driving understanding, scene perception, and action planning, which are coordinated through masked joint attention.<n>It achieves state-of-the-art performance in open-loop evaluation on nuScenes and closed-loop evaluation on Bench2Drive.
  arXiv  Detail & Related papers  (2026-04-02T15:48:45Z)
• Heterogeneous Self-Play for Realistic Highway Traffic Simulation [42.67401323169807]
  PHASE, Policy for Heterogeneous Agent Self-play on Expressway, is a context-aware self-play framework.<n>It supports different vehicle profiles, for example, passenger cars and articulated trailer trucks, within a single policy.<n>It achieves a 96.3% success rate and reduces ADE/FDE from 6.57/12.07 m to 2.44/5.25 m relative to a prior self-play baseline.
  arXiv  Detail & Related papers  (2026-03-31T19:39:29Z)
• Comparative Analysis of Patch Attack on VLM-Based Autonomous Driving Architectures [8.340004072808005]
  This paper presents a systematic framework for comparative adversarial evaluation across three VLM architectures: Dolphins, OmniDrive ( Omni-L), and LeapVAD.<n>Using black-box optimization with semantic homogenization for fair comparison, we evaluate physically realizable patch attacks in CARLA simulation.<n>Results reveal severe vulnerabilities across all architectures, sustained multi-frame failures, and critical object detection degradation.
  arXiv  Detail & Related papers  (2026-03-09T20:04:13Z)
• SAVANT: Semantic Analysis with Vision-Augmented Anomaly deTection [6.806105013817923]
  SAVANT is a structured reasoning framework that achieves high accuracy and recall in detecting anomalous driving scenarios.<n>By automatically labeling over 9,640 real-world images with high accuracy, SAVANT addresses the critical data scarcity problem in anomaly detection.
  arXiv  Detail & Related papers  (2025-10-20T19:14:29Z)
• Natural Reflection Backdoor Attack on Vision Language Model for Autonomous Driving [55.96227460521096]
  Vision-Language Models (VLMs) have been integrated into autonomous driving systems to enhance reasoning capabilities.<n>We propose a natural reflection-based backdoor attack targeting VLM systems in autonomous driving scenarios.<n>Our findings uncover a new class of attacks that exploit the stringent real-time requirements of autonomous driving.
  arXiv  Detail & Related papers  (2025-05-09T20:28:17Z)
• OmniDrive: A Holistic Vision-Language Dataset for Autonomous Driving with Counterfactual Reasoning [68.45848423501927]
  We propose a holistic vision-language dataset that aligns agent models with 3D driving tasks through counterfactual reasoning.<n>Our approach enhances decision-making by evaluating potential scenarios and their outcomes, similar to human drivers considering alternative actions.
  arXiv  Detail & Related papers  (2025-04-06T03:54:21Z)
• Black-Box Adversarial Attack on Vision Language Models for Autonomous Driving [65.61999354218628]
  We take the first step toward designing black-box adversarial attacks specifically targeting vision-language models (VLMs) in autonomous driving systems.<n>We propose Cascading Adversarial Disruption (CAD), which targets low-level reasoning breakdown by generating and injecting semantics.<n>We present Risky Scene Induction, which addresses dynamic adaptation by leveraging a surrogate VLM to understand and construct high-level risky scenarios.
  arXiv  Detail & Related papers  (2025-01-23T11:10:02Z)
• OmniDrive: A Holistic Vision-Language Dataset for Autonomous Driving with Counterfactual Reasoning [68.45848423501927]
  We propose a holistic vision-language dataset that aligns agent models with 3D driving tasks through counterfactual reasoning.<n>Our approach enhances decision-making by evaluating potential scenarios and their outcomes, similar to human drivers considering alternative actions.
  arXiv  Detail & Related papers  (2024-05-02T17:59:24Z)
• Reinforcement Learning with Latent State Inference for Autonomous On-ramp Merging under Observation Delay [6.0111084468944]
  We introduce the Lane-keeping, Lane-changing with Latent-state Inference and Safety Controller (L3IS) agent. L3IS is designed to perform the on-ramp merging task safely without comprehensive knowledge about surrounding vehicles' intents or driving styles. We present an augmentation of this agent called AL3IS that accounts for observation delays, allowing the agent to make more robust decisions in real-world environments.
  arXiv  Detail & Related papers  (2024-03-18T15:02:46Z)
• Detecting 32 Pedestrian Attributes for Autonomous Vehicles [103.87351701138554]
  In this paper, we address the problem of jointly detecting pedestrians and recognizing 32 pedestrian attributes. We introduce a Multi-Task Learning (MTL) model relying on a composite field framework, which achieves both goals in an efficient way. We show competitive detection and attribute recognition results, as well as a more stable MTL training.
  arXiv  Detail & Related papers  (2020-12-04T15:10:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.