AUTOPILOT VQA: Benchmarking Vision-Language Models for Incident-Centric Dashcam Understanding

• URL: http://arxiv.org/abs/2607.08745v1
• Date: Thu, 09 Jul 2026 17:46:24 GMT
• Title: AUTOPILOT VQA: Benchmarking Vision-Language Models for Incident-Centric Dashcam Understanding
• Abstract summary: We present AUTOPILOT-VQA, an incident-centric visual question answering benchmark for dashcam video understanding.<n>The dataset evaluates different systems through structured questions designed around real-world driving incidents and near-incidents.<n>Our benchmark support developments for more interpretable, robust, and safety-conscious vision-language systems for real-world autonomous driving.
• Score: 12.229867730747996
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent advances in Vision-Language Models, Large Language Models, and Multimodal Large Language Models have improved autonomous driving tasks such as scene understanding, decision making, trajectory prediction, and visual question answering. However, evaluating whether these models can reliably reason about safety-critical incidents remains challenging. To address this gap, we present AUTOPILOT-VQA, an incident-centric visual question answering benchmark for dashcam video understanding. The dataset evaluates different systems through structured questions designed around real-world driving incidents and near-incidents. The benchmark covers diverse safety-relevant categories, including weather and lighting conditions, traffic environment, road layout, road surface state, signage, involved entities, accident occurrence, impact location, and avoidability-related reasoning. By requiring models to answer grounded questions about both contextual scene properties and event-level incident details, AUTOPILOT-VQA moves beyond object recognition toward temporally grounded, safety-aware reasoning. The dataset is released as part of the AUTOPILOT CVPR 2026 competition and provides a standardized benchmark for assessing the reliability of autonomous driving systems in different scenarios. Our benchmark support developments for more interpretable, robust, and safety-conscious vision-language systems for real-world autonomous driving.

Related papers

• AD4AD: Benchmarking Visual Anomaly Detection Models for Safer Autonomous Driving [10.121649137872298]
  Failures in autonomous driving translate directly into physical risk for passengers, pedestrians, and other road users.<n>Visual Anomaly Detection (VAD) enables the identification of anomalous objects not present during training.<n>VAD models produce pixel-level anomaly maps that can guide driver attention to specific regions of concern.
  arXiv  Detail & Related papers  (2026-04-16T17:54:53Z)
• CrashSight: A Phase-Aware, Infrastructure-Centric Video Benchmark for Traffic Crash Scene Understanding and Reasoning [27.23760411917563]
  We present textbfCrashSight, a vision-language benchmark for roadway crash understanding using real-world roadside camera data.<n>The dataset comprises 250 crash videos, annotated with 13K multiple-choice question-answer pairs organized under a two-tier taxonomy.<n>We benchmark 8 state-of-the-art VLMs and show that, despite strong scene description capabilities, current models struggle with temporal and causal reasoning in safety-critical scenarios.
  arXiv  Detail & Related papers  (2026-04-09T16:52:04Z)
• SAVeS: Steering Safety Judgments in Vision-Language Models via Semantic Cues [50.928203120795786]
  We study whether multimodal safety behavior in vision-language models can be steered by simple semantic cues.<n>We introduce a semantic steering framework that applies controlled textual, visual, and cognitive interventions without changing the underlying scene content.
  arXiv  Detail & Related papers  (2026-03-19T16:18:00Z)
• Vision and language: Novel Representations and Artificial intelligence for Driving Scene Safety Assessment and Autonomous Vehicle Planning [2.1379801460200416]
  Vision-language models (VLMs) have emerged as powerful representation learning systems that align visual observations with natural language concepts.<n>This paper investigates how vision-language representations support driving scene safety assessment and decision-making when integrated into perception, prediction, and planning pipelines.
  arXiv  Detail & Related papers  (2026-02-07T20:04:21Z)
• AccidentBench: Benchmarking Multimodal Understanding and Reasoning in Vehicle Accidents and Beyond [101.20320617562321]
  AccidentBench is a large-scale benchmark that combines vehicle accident scenarios with Beyond domains.<n>The benchmark contains approximately 2000 videos and over 19000 human-annotated question-answer pairs.
  arXiv  Detail & Related papers  (2025-09-30T17:59:13Z)
• Natural Reflection Backdoor Attack on Vision Language Model for Autonomous Driving [55.96227460521096]
  Vision-Language Models (VLMs) have been integrated into autonomous driving systems to enhance reasoning capabilities.<n>We propose a natural reflection-based backdoor attack targeting VLM systems in autonomous driving scenarios.<n>Our findings uncover a new class of attacks that exploit the stringent real-time requirements of autonomous driving.
  arXiv  Detail & Related papers  (2025-05-09T20:28:17Z)
• Are VLMs Ready for Autonomous Driving? An Empirical Study from the Reliability, Data, and Metric Perspectives [56.528835143531694]
  We introduce DriveBench, a benchmark dataset designed to evaluate Vision-Language Models (VLMs)<n>Our findings reveal that VLMs often generate plausible responses derived from general knowledge or textual cues rather than true visual grounding.<n>We propose refined evaluation metrics that prioritize robust visual grounding and multi-modal understanding.
  arXiv  Detail & Related papers  (2025-01-07T18:59:55Z)
• CRASH: Crash Recognition and Anticipation System Harnessing with Context-Aware and Temporal Focus Attentions [13.981748780317329]
  Accurately and promptly predicting accidents among surrounding traffic agents from camera footage is crucial for the safety of autonomous vehicles (AVs) This study introduces a novel accident anticipation framework for AVs, termed CRASH. It seamlessly integrates five components: object detector, feature extractor, object-aware module, context-aware module, and multi-layer fusion. Our model surpasses existing top baselines in critical evaluation metrics like Average Precision (AP) and mean Time-To-Accident (mTTA)
  arXiv  Detail & Related papers  (2024-07-25T04:12:49Z)
• Leveraging Driver Field-of-View for Multimodal Ego-Trajectory Prediction [69.29802752614677]
  RouteFormer is a novel ego-trajectory prediction network combining GPS data, environmental context, and the driver's field-of-view.<n>To tackle data scarcity and enhance diversity, we introduce GEM, a dataset of urban driving scenarios enriched with synchronized driver field-of-view and gaze data.
  arXiv  Detail & Related papers  (2023-12-13T23:06:30Z)
• Camera-Radar Perception for Autonomous Vehicles and ADAS: Concepts, Datasets and Metrics [77.34726150561087]
  This work aims to carry out a study on the current scenario of camera and radar-based perception for ADAS and autonomous vehicles. Concepts and characteristics related to both sensors, as well as to their fusion, are presented. We give an overview of the Deep Learning-based detection and segmentation tasks, and the main datasets, metrics, challenges, and open questions in vehicle perception.
  arXiv  Detail & Related papers  (2023-03-08T00:48:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.