Evaluating Small Vision-Language Models on Distance-Dependent Traffic Perception

• URL: http://arxiv.org/abs/2510.08352v1
• Date: Thu, 09 Oct 2025 15:38:41 GMT
• Title: Evaluating Small Vision-Language Models on Distance-Dependent Traffic Perception
• Authors: Nikos Theodoridis, Tim Brophy, Reenu Mohandas, Ganesh Sistu, Fiachra Collins, Anthony Scanlan, Ciaran Eising,
• Abstract summary: We introduce Distance-Annotated Traffic Perception Question Answering (DTPQA) benchmark.<n>First Visual Question Answering (VQA) benchmark focused solely on perception-based questions in traffic scenes.<n>We evaluate several state-of-the-art (SOTA) small Vision-Language Models (VLMs) on DTPQA.
• Score: 0.7644902597398215
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision-Language Models (VLMs) are becoming increasingly powerful, demonstrating strong performance on a variety of tasks that require both visual and textual understanding. Their strong generalisation abilities make them a promising component for automated driving systems, which must handle unexpected corner cases. However, to be trusted in such safety-critical applications, a model must first possess a reliable perception system. Moreover, since critical objects and agents in traffic scenes are often at a distance, we require systems that are not "shortsighted", i.e., systems with strong perception capabilities at both close (up to 20 meters) and long (30+ meters) range. With this in mind, we introduce Distance-Annotated Traffic Perception Question Answering (DTPQA), the first Visual Question Answering (VQA) benchmark focused solely on perception-based questions in traffic scenes, enriched with distance annotations. By excluding questions that require reasoning, we ensure that model performance reflects perception capabilities alone. Since automated driving hardware has limited processing power and cannot support large VLMs, our study centers on smaller VLMs. More specifically, we evaluate several state-of-the-art (SOTA) small VLMs on DTPQA and show that, despite the simplicity of the questions, these models significantly underperform compared to humans (~60% average accuracy for the best-performing small VLM versus ~85% human performance). However, it is important to note that the human sample size was relatively small, which imposes statistical limitations. We also identify specific perception tasks, such as distinguishing left from right, that remain particularly challenging for these models.

Related papers

• Descriptor: Distance-Annotated Traffic Perception Question Answering (DTPQA) [0.7644902597398215]
  Distance-Annotated Traffic Perception Question Answering (DTPQA) is a Visual Question Answering (VQA) benchmark designed for this purpose.<n>It can be used to evaluate the perception systems of Vision-Language Models (VLMs) in traffic scenarios using trivial yet crucial questions relevant to driving decisions.<n>It consists of two parts: a synthetic benchmark (DTP-Synthetic) created using a simulator, and a real-world benchmark (DTP-Real) built on top of existing images of real traffic scenes.
  arXiv  Detail & Related papers  (2025-11-17T14:12:22Z)
• Are VLMs Ready for Lane Topology Awareness in Autonomous Driving? [17.325365876089602]
  Vision-Language Models (VLMs) have recently shown remarkable progress in multimodal reasoning, yet their applications in autonomous driving remain limited.<n>In this work, we systematically evaluate VLMs' capabilities in road topology understanding.
  arXiv  Detail & Related papers  (2025-09-20T12:02:39Z)
• BetterCheck: Towards Safeguarding VLMs for Automotive Perception Systems [1.701722696403793]
  Large language models (LLMs) are growingly extended to process multimodal data such as text and video simultaneously.<n>When non-restricted, LLMs and vision language models (VLMs) show impressive performance to describe even complex traffic situations.<n>VLMs are prone to hallucination, which means to either potentially not seeing traffic agents who are present in a situation, or to seeing traffic agents who are not there in reality.
  arXiv  Detail & Related papers  (2025-07-23T17:32:17Z)
• Response Wide Shut? Surprising Observations in Basic Vision Language Model Capabilities [54.94982467313341]
  Vision-language Models (VLMs) have emerged as general-purpose tools for addressing a variety of complex computer vision problems.<n>We set out to understand the limitations of SoTA VLMs on fundamental visual tasks by constructing a series of tests that probe which components of design, specifically, may be lacking.
  arXiv  Detail & Related papers  (2025-07-10T15:26:41Z)
• Natural Reflection Backdoor Attack on Vision Language Model for Autonomous Driving [55.96227460521096]
  Vision-Language Models (VLMs) have been integrated into autonomous driving systems to enhance reasoning capabilities.<n>We propose a natural reflection-based backdoor attack targeting VLM systems in autonomous driving scenarios.<n>Our findings uncover a new class of attacks that exploit the stringent real-time requirements of autonomous driving.
  arXiv  Detail & Related papers  (2025-05-09T20:28:17Z)
• DriveLMM-o1: A Step-by-Step Reasoning Dataset and Large Multimodal Model for Driving Scenario Understanding [76.3876070043663]
  We propose DriveLMM-o1, a dataset and benchmark designed to advance step-wise visual reasoning for autonomous driving.<n>Our benchmark features over 18k VQA examples in the training set and more than 4k in the test set, covering diverse questions on perception, prediction, and planning.<n>Our model achieves a +7.49% gain in final answer accuracy, along with a 3.62% improvement in reasoning score over the previous best open-source model.
  arXiv  Detail & Related papers  (2025-03-13T17:59:01Z)
• Are VLMs Ready for Autonomous Driving? An Empirical Study from the Reliability, Data, and Metric Perspectives [56.528835143531694]
  We introduce DriveBench, a benchmark dataset designed to evaluate Vision-Language Models (VLMs)<n>Our findings reveal that VLMs often generate plausible responses derived from general knowledge or textual cues rather than true visual grounding.<n>We propose refined evaluation metrics that prioritize robust visual grounding and multi-modal understanding.
  arXiv  Detail & Related papers  (2025-01-07T18:59:55Z)
• VHELM: A Holistic Evaluation of Vision Language Models [75.88987277686914]
  We present the Holistic Evaluation of Vision Language Models (VHELM) VHELM aggregates various datasets to cover one or more of the 9 aspects: visual perception, knowledge, reasoning, bias, fairness, multilinguality, robustness, toxicity, and safety. Our framework is designed to be lightweight and automatic so that evaluation runs are cheap and fast.
  arXiv  Detail & Related papers  (2024-10-09T17:46:34Z)
• Response Wide Shut: Surprising Observations in Basic Vision Language Model Capabilities [30.176918208200604]
  Vision-Language Models (VLMs) have emerged as general purpose tools for addressing a variety of complex computer vision problems. These models have been shown to be highly capable, but also lacking some basic visual understanding skills. This paper sets out to understand the limitations of SoTA VLMs on fundamental visual tasks.
  arXiv  Detail & Related papers  (2024-08-13T08:26:32Z)
• Revisiting Few-Shot Object Detection with Vision-Language Models [49.79495118650838]
  We revisit the task of few-shot object detection (FSOD) in the context of recent foundational vision-language models (VLMs) We propose Foundational FSOD, a new benchmark protocol that evaluates detectors pre-trained on any external data. We discuss our recent CVPR 2024 Foundational FSOD competition and share insights from the community.
  arXiv  Detail & Related papers  (2023-12-22T07:42:00Z)
• DriveLM: Driving with Graph Visual Question Answering [57.51930417790141]
  We study how vision-language models (VLMs) trained on web-scale data can be integrated into end-to-end driving systems.<n>We propose a VLM-based baseline approach (DriveLM-Agent) for jointly performing Graph VQA and end-to-end driving.
  arXiv  Detail & Related papers  (2023-12-21T18:59:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.