How Well Do Vision-Language Models Understand Sequential Driving Scenes? A Sensitivity Study

• URL: http://arxiv.org/abs/2604.06750v1
• Date: Wed, 08 Apr 2026 07:14:55 GMT
• Title: How Well Do Vision-Language Models Understand Sequential Driving Scenes? A Sensitivity Study
• Abstract summary: VENUSS is a framework for systematic sensitivity analysis of VLM performance on sequential driving scenes.<n>It extracts temporal sequences from driving videos, and generates structured evaluations across custom categories.<n>We show how even top models achieve only 57% accuracy, not matching human performance in similar constraints.
• Score: 2.6954666679827137
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision-Language Models (VLMs) are increasingly proposed for autonomous driving tasks, yet their performance on sequential driving scenes remains poorly characterized, particularly regarding how input configurations affect their capabilities. We introduce VENUSS (VLM Evaluation oN Understanding Sequential Scenes), a framework for systematic sensitivity analysis of VLM performance on sequential driving scenes, establishing baselines for future research. Building upon existing datasets, VENUSS extracts temporal sequences from driving videos, and generates structured evaluations across custom categories. By comparing 25+ existing VLMs across 2,600+ scenarios, we reveal how even top models achieve only 57% accuracy, not matching human performance in similar constraints (65%) and exposing significant capability gaps. Our analysis shows that VLMs excel with static object detection but struggle with understanding the vehicle dynamics and temporal relations. VENUSS offers the first systematic sensitivity analysis of VLMs focused on how input image configurations - resolution, frame count, temporal intervals, spatial layouts, and presentation modes - affect performance on sequential driving scenes. Supplementary material available at https://V3NU55.github.io

Related papers

• Mask What Matters: Saliency-Guided Video Self-Supervised Learning for Autonomous Driving [57.56568641990586]
  We introduce V-JEPA4A, a domain-specialized variant of V-JEPA for autonomous driving that is pre-trained on publicly available driving videos with a novel saliency-driven masking policy.<n>V-JEPA4A reduces identity switches on BDD100k MOT by 25% over V-JEPA with random masking, achieves 73.2 mIoU on City, and 3.75 RMSE on KITTI-2015 depth, while incurring only 14% additional pre-training overhead.
  arXiv  Detail & Related papers  (2026-08-17T22:39:49Z)
• MVPruner: Dynamic Token Pruning for Accelerating Multi-view Vision-Language Models in Autonomous Driving [32.02133342612463]
  Vision-Language Models (VLMs) improve generalization and interpretability in autonomous driving.<n>Existing token pruning methods employ fixed pruning rate allocation and static importance metrics.<n>We propose MVPruner, a two-stage adaptive token pruning method that aligns pruning behavior with the model's dynamic information requirements.
  arXiv  Detail & Related papers  (2026-06-26T02:33:20Z)
• DRIVESPATIAL: A Benchmark for Spatiotemporal Intelligence in VLMs for Autonomous Driving [15.87281823665284]
  We introduce DriveSpatial, a benchmark of 15.6K-verified QA pairs across 20 tasks from five large-scale autonomous driving datasets.<n>Unlike prior benchmarks DriveSpatial is generated from a dynamic multi-language scene graph that encodes object states, spatial relations, interactions, camera visibility, and temporal correspondences.<n>We find that the strongest model trails humans by 28.4 points, with Cognitive Scene Construction emerging as the key bottleneck.
  arXiv  Detail & Related papers  (2026-05-22T02:52:06Z)
• OneDrive: Unified Multi-Paradigm Driving with Vision-Language-Action Models [69.2503510410147]
  We present a unified autonomous driving framework built upon a pretrained VLM.<n>We demonstrate that pretrained VLM attention exhibits strong transferability beyond pure language modeling.<n>Experiments on end-to-end autonomous driving benchmarks demonstrate state-of-the-art performance.
  arXiv  Detail & Related papers  (2026-04-20T07:50:00Z)
• iFinder: Structured Zero-Shot Vision-Based LLM Grounding for Dash-Cam Video Reasoning [51.15353027471834]
  iFinder is a semantic grounding framework that translates dash-cam videos into a hierarchical, interpretable data structure for large language models.<n>iFinder operates as a training-free pipeline that employs pretrained vision models to extract critical cues.<n>It significantly outperforms end-to-end V-VLMs on four zero-shot driving benchmarks.
  arXiv  Detail & Related papers  (2025-09-23T20:25:53Z)
• FastDriveVLA: Efficient End-to-End Driving via Plug-and-Play Reconstruction-based Token Pruning [75.80110543049783]
  We propose FastDriveVLA, a reconstruction-based vision token pruning framework for autonomous driving.<n>A novel foreground adversarial-background reconstruction strategy is designed to train ReconPruner for the visual encoder of VLA models.<n>Our approach achieves state-of-the-art results on the nuScenes open-loop planning benchmark across different pruning ratios.
  arXiv  Detail & Related papers  (2025-07-31T07:55:56Z)
• STSBench: A Spatio-temporal Scenario Benchmark for Multi-modal Large Language Models in Autonomous Driving [16.602141801221364]
  STSBench is a framework to benchmark holistic understanding of vision-language models (VLMs) for autonomous driving.<n>The benchmark features 43 diverse scenarios spanning multiple views, resulting in 971 human-verified multiple-choice questions.<n>A thorough evaluation uncovers shortcomings in existing models' ability to reason about fundamental traffic dynamics in complex environments.
  arXiv  Detail & Related papers  (2025-06-06T16:25:22Z)
• Are Vision LLMs Road-Ready? A Comprehensive Benchmark for Safety-Critical Driving Video Understanding [10.242043337117005]
  Vision Large Language Models (VLLMs) have demonstrated impressive capabilities in general visual tasks such as image captioning and visual question answering.<n>However, their effectiveness in specialized, safety-critical domains like autonomous driving remains largely unexplored.<n>We introduce DVBench, a pioneering benchmark designed to evaluate the performance of VLLMs in understanding safety-critical driving videos.
  arXiv  Detail & Related papers  (2025-04-20T07:50:44Z)
• Are VLMs Ready for Autonomous Driving? An Empirical Study from the Reliability, Data, and Metric Perspectives [56.528835143531694]
  We introduce DriveBench, a benchmark dataset designed to evaluate Vision-Language Models (VLMs)<n>Our findings reveal that VLMs often generate plausible responses derived from general knowledge or textual cues rather than true visual grounding.<n>We propose refined evaluation metrics that prioritize robust visual grounding and multi-modal understanding.
  arXiv  Detail & Related papers  (2025-01-07T18:59:55Z)
• LaVida Drive: Vision-Text Interaction VLM for Autonomous Driving with Token Selection, Recovery and Enhancement [4.534832757549232]
  We introduce LaVida Drive, a novel and efficient VQA framework for autonomous driving.<n>LaVida Drive seamlessly integrates temporal data while maintaining high-resolution inputs for detailed visual perception.<n>It optimize spatial processing by retaining high-resolution data for intricate details and using lower-resolution inputs for temporal analysis.
  arXiv  Detail & Related papers  (2024-11-20T02:14:07Z)
• Probing Multimodal LLMs as World Models for Driving [72.18727651074563]
  We look at the application of Multimodal Large Language Models (MLLMs) in autonomous driving. Despite advances in models like GPT-4o, their performance in complex driving environments remains largely unexplored.
  arXiv  Detail & Related papers  (2024-05-09T17:52:42Z)
• DriveLM: Driving with Graph Visual Question Answering [57.51930417790141]
  We study how vision-language models (VLMs) trained on web-scale data can be integrated into end-to-end driving systems.<n>We propose a VLM-based baseline approach (DriveLM-Agent) for jointly performing Graph VQA and end-to-end driving.
  arXiv  Detail & Related papers  (2023-12-21T18:59:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.