VRU-Accident: A Vision-Language Benchmark for Video Question Answering and Dense Captioning for Accident Scene Understanding

• URL: http://arxiv.org/abs/2507.09815v2
• Date: Tue, 22 Jul 2025 17:15:14 GMT
• Title: VRU-Accident: A Vision-Language Benchmark for Video Question Answering and Dense Captioning for Accident Scene Understanding
• Abstract summary: multimodal large language models (MLLMs) have shown promise in enhancing scene understanding and decision making in autonomous vehicles.<n>We present VRU-Accident, a vision-language benchmark designed to evaluate MLLMs in high-risk traffic scenarios involving VRUs.<n>Unlike prior works, our benchmark focuses explicitly on VRU-vehicle accidents, providing rich, fine-grained annotations that capture both spatial-temporal dynamics and causal semantics of accidents.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Ensuring the safety of vulnerable road users (VRUs), such as pedestrians and cyclists, is a critical challenge for autonomous driving systems, as crashes involving VRUs often result in severe or fatal consequences. While multimodal large language models (MLLMs) have shown promise in enhancing scene understanding and decision making in autonomous vehicles, there is currently no standardized benchmark to quantitatively evaluate their reasoning abilities in complex, safety-critical scenarios involving VRUs. To address this gap, we present VRU-Accident, a large-scale vision-language benchmark designed to evaluate MLLMs in high-risk traffic scenarios involving VRUs. VRU-Accident comprises 1K real-world dashcam accident videos, annotated with 6K multiple-choice question-answer pairs across six safety-critical categories (with 24K candidate options and 3.4K unique answer choices), as well as 1K dense scene descriptions. Unlike prior works, our benchmark focuses explicitly on VRU-vehicle accidents, providing rich, fine-grained annotations that capture both spatial-temporal dynamics and causal semantics of accidents. To assess the current landscape of MLLMs, we conduct a comprehensive evaluation of 17 state-of-the-art models on the multiple-choice VQA task and on the dense captioning task. Our findings reveal that while MLLMs perform reasonably well on visually grounded attributes, they face significant challenges in reasoning and describing accident causes, types, and preventability.

Related papers

• CAViAR: A Causal Video Dataset for Fine-Grained Accident Reasoning in Real-World Scenarios [10.195587813770075]
  We introduce CAViAR (Causal Accident Video and Incident Analysis Repository), a human-annotated dashcam benchmark comprising 2,249 real-world accident videos.<n>Each video is annotated with structured labels spanning environmental conditions, accident type, causal explanation, apparent At-Fault Agent, affected agent, and apparent rule-violation category.<n>We benchmark state-of-the-art vision-language models (VLMs), including Cosmos-Reason2, Qwen3-VL, and InternVL3.
  arXiv  Detail & Related papers  (2026-08-19T18:51:21Z)
• Technical Report on the CVPR 2026@AdvML Workshop Challenge [212.8018524968403]
  Vision-language agents (VLAs) are increasingly used to interpret complex driving scenes and support safety-critical reasoning.<n>This report presents the CVPR 2026@AdvML Workshop Challenge on adversarial multimodal attacks against autonomous-driving VLAs.
  arXiv  Detail & Related papers  (2026-07-13T13:42:48Z)
• AUTOPILOT VQA: Benchmarking Vision-Language Models for Incident-Centric Dashcam Understanding [12.229867730747996]
  We present AUTOPILOT-VQA, an incident-centric visual question answering benchmark for dashcam video understanding.<n>The dataset evaluates different systems through structured questions designed around real-world driving incidents and near-incidents.<n>Our benchmark support developments for more interpretable, robust, and safety-conscious vision-language systems for real-world autonomous driving.
  arXiv  Detail & Related papers  (2026-07-09T17:46:24Z)
• SAVeS: Steering Safety Judgments in Vision-Language Models via Semantic Cues [50.928203120795786]
  We study whether multimodal safety behavior in vision-language models can be steered by simple semantic cues.<n>We introduce a semantic steering framework that applies controlled textual, visual, and cognitive interventions without changing the underlying scene content.
  arXiv  Detail & Related papers  (2026-03-19T16:18:00Z)
• VLM-AutoDrive: Post-Training Vision-Language Models for Safety-Critical Autonomous Driving Events [25.118024547769895]
  We introduce VLM-AutoDrive, a modular framework for adapting pretrained Vision-Language Models to high-fidelity anomaly detection.<n>Off-the-shelf VLMs such as NVIDIA's Cosmos-1 7B (CR1) exhibit near-zero Collision recall in zero-shot settings.<n>VLM-AutoDrive offers a scalable recipe for adapting general-purpose VLMs to safety-critical, temporally localized perception tasks.
  arXiv  Detail & Related papers  (2026-03-18T18:23:34Z)
• Towards Safer Mobile Agents: Scalable Generation and Evaluation of Diverse Scenarios for VLMs [10.48956192789531]
  Vision Language Models (VLMs) are increasingly deployed in autonomous vehicles and mobile systems.<n>Current benchmarks inadequately cover diverse hazardous situations, especially anomalous scenarios with.<n>temporal dynamics.<n>We introduce textbfHazardForge, a scalable pipeline that leverages image editing models to generate.<n>scenarios with layout decision algorithms, and validation modules.
  arXiv  Detail & Related papers  (2026-01-13T11:55:31Z)
• WaymoQA: A Multi-View Visual Question Answering Dataset for Safety-Critical Reasoning in Autonomous Driving [33.850069933308994]
  High-level reasoning in safety-critical scenarios remains a major challenge.<n>We define Safety-Critical Reasoning as a new task that leverages multi-view inputs to address this challenge.<n>We introduceQA, a dataset of 35,000 human-annotated question-answer pairs covering complex, high-risk driving scenarios.
  arXiv  Detail & Related papers  (2025-11-25T07:47:27Z)
• AccidentBench: Benchmarking Multimodal Understanding and Reasoning in Vehicle Accidents and Beyond [101.20320617562321]
  AccidentBench is a large-scale benchmark that combines vehicle accident scenarios with Beyond domains.<n>The benchmark contains approximately 2000 videos and over 19000 human-annotated question-answer pairs.
  arXiv  Detail & Related papers  (2025-09-30T17:59:13Z)
• DRAMA-X: A Fine-grained Intent Prediction and Risk Reasoning Benchmark For Driving [5.362063089413001]
  No existing benchmark evaluates multi-class intent prediction in safety-critical situations.<n>We introduce DRAMA-X, a fine-grained benchmark constructed from the DRAMA dataset.<n>We propose SGG-Intent, a lightweight, training-free framework that mirrors the ego vehicle's reasoning pipeline.
  arXiv  Detail & Related papers  (2025-06-21T05:01:42Z)
• Natural Reflection Backdoor Attack on Vision Language Model for Autonomous Driving [55.96227460521096]
  Vision-Language Models (VLMs) have been integrated into autonomous driving systems to enhance reasoning capabilities.<n>We propose a natural reflection-based backdoor attack targeting VLM systems in autonomous driving scenarios.<n>Our findings uncover a new class of attacks that exploit the stringent real-time requirements of autonomous driving.
  arXiv  Detail & Related papers  (2025-05-09T20:28:17Z)
• Are Vision LLMs Road-Ready? A Comprehensive Benchmark for Safety-Critical Driving Video Understanding [10.242043337117005]
  Vision Large Language Models (VLLMs) have demonstrated impressive capabilities in general visual tasks such as image captioning and visual question answering.<n>However, their effectiveness in specialized, safety-critical domains like autonomous driving remains largely unexplored.<n>We introduce DVBench, a pioneering benchmark designed to evaluate the performance of VLLMs in understanding safety-critical driving videos.
  arXiv  Detail & Related papers  (2025-04-20T07:50:44Z)
• Road Rage Reasoning with Vision-language Models (VLMs): Task Definition and Evaluation Dataset [4.357836359387452]
  Road rage, triggered by driving-related stimuli such as traffic congestion and aggressive driving, poses a significant threat to road safety.<n>Previous research on road rage regulation has primarily focused on response suppression, lacking proactive prevention capabilities.<n>With the advent of Vision-Language Models (VLMs), it has become possible to reason about trigger events visually and then engage in dialog-based comforting before drivers' anger escalates.
  arXiv  Detail & Related papers  (2025-03-14T12:18:11Z)
• Black-Box Adversarial Attack on Vision Language Models for Autonomous Driving [65.61999354218628]
  We take the first step toward designing black-box adversarial attacks specifically targeting vision-language models (VLMs) in autonomous driving systems.<n>We propose Cascading Adversarial Disruption (CAD), which targets low-level reasoning breakdown by generating and injecting semantics.<n>We present Risky Scene Induction, which addresses dynamic adaptation by leveraging a surrogate VLM to understand and construct high-level risky scenarios.
  arXiv  Detail & Related papers  (2025-01-23T11:10:02Z)
• Are VLMs Ready for Autonomous Driving? An Empirical Study from the Reliability, Data, and Metric Perspectives [56.528835143531694]
  We introduce DriveBench, a benchmark dataset designed to evaluate Vision-Language Models (VLMs)<n>Our findings reveal that VLMs often generate plausible responses derived from general knowledge or textual cues rather than true visual grounding.<n>We propose refined evaluation metrics that prioritize robust visual grounding and multi-modal understanding.
  arXiv  Detail & Related papers  (2025-01-07T18:59:55Z)
• DAVE: Diverse Atomic Visual Elements Dataset with High Representation of Vulnerable Road Users in Complex and Unpredictable Environments [60.69159598130235]
  We present a new dataset, DAVE, designed for evaluating perception methods with high representation of Vulnerable Road Users (VRUs)<n>DAVE is a manually annotated dataset encompassing 16 diverse actor categories (spanning animals, humans, vehicles, etc.) and 16 action types (complex and rare cases like cut-ins, zigzag movement, U-turn, etc.)<n>Our experiments show that existing methods suffer degradation in performance when evaluated on DAVE, highlighting its benefit for future video recognition research.
  arXiv  Detail & Related papers  (2024-12-28T06:13:44Z)
• ScVLM: Enhancing Vision-Language Model for Safety-Critical Event Understanding [5.914751204116458]
  We introduce ScVLM, a novel hybrid methodology that integrates supervised and contrastive learning techniques to classify severity and types of SCEs.<n>The proposed approach is trained on and evaluated by more than 8,600 SCEs from the Second Strategic Highway Research Program Naturalistic Driving Study dataset.
  arXiv  Detail & Related papers  (2024-10-01T18:10:23Z)
• Abductive Ego-View Accident Video Understanding for Safe Driving Perception [75.60000661664556]
  We present MM-AU, a novel dataset for Multi-Modal Accident video Understanding. MM-AU contains 11,727 in-the-wild ego-view accident videos, each with temporally aligned text descriptions. We present an Abductive accident Video understanding framework for Safe Driving perception (AdVersa-SD)
  arXiv  Detail & Related papers  (2024-03-01T10:42:52Z)
• DeepAccident: A Motion and Accident Prediction Benchmark for V2X Autonomous Driving [76.29141888408265]
  We propose a large-scale dataset containing diverse accident scenarios that frequently occur in real-world driving. The proposed DeepAccident dataset includes 57K annotated frames and 285K annotated samples, approximately 7 times more than the large-scale nuScenes dataset.
  arXiv  Detail & Related papers  (2023-04-03T17:37:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.