Does Visual Information Play a Decisive Role in Vision-Language-Action Model Driving Behavior?

• URL: http://arxiv.org/abs/2605.31041v1
• Date: Fri, 29 May 2026 09:18:32 GMT
• Title: Does Visual Information Play a Decisive Role in Vision-Language-Action Model Driving Behavior?
• Abstract summary: Vision-Language-Action (VLA) models have demonstrated promising capability in autonomous driving.<n>How current VLA-based driving behavior is grounded in visual information remains poorly understood.<n>We introduce a structured multi-level visual framework to analyze visual-behavior dependency in VLA-based driving models systematically.
• Score: 6.847591547447323
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision-Language-Action (VLA) models have demonstrated promising capability in autonomous driving, highlighting the potential of unified multimodal architectures for jointly modeling perception and planning. However, how current VLA-based driving behavior is grounded in visual information remains poorly understood. Existing evaluation protocols mainly focus on aggregate performance metrics, lacking structured and practical diagnostics to quantify visual-behavior dependency. In this work, we introduce a structured multi-level visual perturbation framework to analyze visual-behavior dependency in VLA-based driving models systematically. The framework organizes controlled visual perturbations along three complementary dimensions: channellevel degradation, information-level disruption, and structurelevel modification. We apply it to VLA-based driving systems and evaluate behavioral responses under both open-loop trajectory prediction and interactive closed-loop safety evaluation. Experimental results reveal evaluation-dependent dependency patterns and uneven visual grounding across abstraction levels. These findings call for more structured analyses and principled design of VLA driving models to better understand how visual information shapes behavior and develop safer, more robust systems.

Related papers

• Explicit Kinematic Guidance from Analytic Concepts for Vision-Language-Action Models [67.98329043920803]
  Current Vision-Language-Action (VLA) models rely mainly on 2D inputs, neglecting the rich object structural information and commonsense knowledge inherent in the 3D physical world.<n>This deficiency restricts their spatial awareness and adaptability for complex, high-precision manipulation.<n>We construct a Concept Expert module for VLA to build executable Analytic Concepts that represent objects as explicit, programmatic blueprints.
  arXiv  Detail & Related papers  (2026-07-29T06:24:25Z)
• A Causality-aware Infer-diagnose-refine Framework for Test-time Modality Adaptation in VLA Models [16.665710235622214]
  We propose an infer-diagnose-refine (IDR) framework to fuse modalities in vision-language-action (VLA) models.<n>IDR first infers actions under factual and counterfactual scenarios of visual observations, and then diagnoses the causal effects of visual observations as the estimated dynamic importance.<n>Experiments on both simulation benchmarks and real-world tasks show improvements in overall performance across multiple VLA backbones.
  arXiv  Detail & Related papers  (2026-07-28T09:56:42Z)
• VLA-Trace: Diagnosing Vision-Language-Action Models through Representation and Behavior Tracing [12.088525050678713]
  Vision-Language-Action (VLA) models transform multimodal knowledge into embodied control.<n>We present VLA-Trace, a progressive diagnostic framework that analyzes VLA models through a unified evidence chain.
  arXiv  Detail & Related papers  (2026-05-28T15:50:56Z)
• Afford-VLA: Action-Aligned Visual Planning via Internalized Affordance [108.46436073194546]
  We argue that effective planning should be local, visually grounded, internally generated, and directly aligned with action.<n>We propose Afford-VLA, a unified framework that internalizes task-conditioned affordance as an explicit visual planning interface within VLA models.
  arXiv  Detail & Related papers  (2026-05-22T20:43:47Z)
• Look Before Acting: Enhancing Vision Foundation Representations for Vision-Language-Action Models [66.96421290733126]
  Vision-Language-Action (VLA) models have emerged as a promising paradigm for robotic manipulation.<n>We propose textbfDeepVision-VLA, built on a textbfVision-Language Mixture-of-Transformers (VL-MoT) framework.<n>DeepVision-VLA outperforms prior state-of-the-art methods by 9.0% and 7.5% on simulated and real-world tasks, respectively.
  arXiv  Detail & Related papers  (2026-03-16T17:59:54Z)
• From Perception to Action: An Interactive Benchmark for Vision Reasoning [51.11355591375073]
  Causal Hierarchy of Actions and Interactions (CHAIN) benchmark designed to evaluate whether models can understand, plan, and execute structured action sequences grounded in physical constraints.<n> CHAIN shifts evaluation from passive perception to active problem solving, spanning tasks such as interlocking mechanical puzzles and 3D stacking and packing.<n>Our results show that top-performing models still struggle to internalize physical structure and causal constraints, often failing to produce reliable long-horizon plans and cannot robustly translate perceived structure into effective actions.
  arXiv  Detail & Related papers  (2026-02-24T15:33:02Z)
• Vision-Language-Action Models for Autonomous Driving: Past, Present, and Future [125.92052530850425]
  Vision-Language-Action (VLA) frameworks integrate perception with language-grounded decision making.<n>VLA frameworks offer a pathway toward more interpretable, generalizable, and human-aligned driving policies.<n>This work aims to establish a coherent foundation for advancing human-compatible autonomous driving systems.
  arXiv  Detail & Related papers  (2025-12-18T16:57:44Z)
• RAD: Retrieval-Augmented Decision-Making of Meta-Actions with Vision-Language Models in Autonomous Driving [10.984203470464687]
  Vision-language models (VLMs) often suffer from limitations such as inadequate spatial perception and hallucination.<n>We propose a retrieval-augmented decision-making (RAD) framework to enhance VLMs' capabilities to reliably generate meta-actions in autonomous driving scenes.<n>We fine-tune VLMs on a dataset derived from the NuScenes dataset to enhance their spatial perception and bird's-eye view image comprehension capabilities.
  arXiv  Detail & Related papers  (2025-03-18T03:25:57Z)
• Application of Vision-Language Model to Pedestrians Behavior and Scene Understanding in Autonomous Driving [5.456780031044544]
  We propose a knowledge distillation method that transfers knowledge from large-scale vision-language foundation models to efficient vision networks.<n>We apply it to pedestrian behavior prediction and scene understanding tasks, achieving promising results in generating more diverse and comprehensive semantic attributes.
  arXiv  Detail & Related papers  (2025-01-12T01:31:07Z)
• Explanatory Model Monitoring to Understand the Effects of Feature Shifts on Performance [61.06245197347139]
  We propose a novel approach to explain the behavior of a black-box model under feature shifts. We refer to our method that combines concepts from Optimal Transport and Shapley Values as Explanatory Performance Estimation.
  arXiv  Detail & Related papers  (2024-08-24T18:28:19Z)
• Reason induced visual attention for explainable autonomous driving [2.090380922731455]
  Deep learning (DL) based computer vision (CV) models are generally considered as black boxes due to poor interpretability. This study is motivated by the need to enhance the interpretability of DL model in autonomous driving. The proposed framework imitates the learning process of human drivers by jointly modeling the visual input (images) and natural language.
  arXiv  Detail & Related papers  (2021-10-11T18:50:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.