WaymoQA: A Multi-View Visual Question Answering Dataset for Safety-Critical Reasoning in Autonomous Driving

• URL: http://arxiv.org/abs/2511.20022v1
• Date: Tue, 25 Nov 2025 07:47:27 GMT
• Title: WaymoQA: A Multi-View Visual Question Answering Dataset for Safety-Critical Reasoning in Autonomous Driving
• Authors: Seungjun Yu, Seonho Lee, Namho Kim, Jaeyo Shin, Junsung Park, Wonjeong Ryu, Raehyuk Jung, Hyunjung Shim,
• Abstract summary: High-level reasoning in safety-critical scenarios remains a major challenge.<n>We define Safety-Critical Reasoning as a new task that leverages multi-view inputs to address this challenge.<n>We introduceQA, a dataset of 35,000 human-annotated question-answer pairs covering complex, high-risk driving scenarios.
• Score: 33.850069933308994
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent advancements in multimodal large language models (MLLMs) have shown strong understanding of driving scenes, drawing interest in their application to autonomous driving. However, high-level reasoning in safety-critical scenarios, where avoiding one traffic risk can create another, remains a major challenge. Such reasoning is often infeasible with only a single front view and requires a comprehensive view of the environment, which we achieve through multi-view inputs. We define Safety-Critical Reasoning as a new task that leverages multi-view inputs to address this challenge. Then, we distill Safety-Critical Reasoning into two stages: first resolve the immediate risk, then mitigate the decision-induced downstream risks. To support this, we introduce WaymoQA, a dataset of 35,000 human-annotated question-answer pairs covering complex, high-risk driving scenarios. The dataset includes multiple-choice and open-ended formats across both image and video modalities. Experiments reveal that existing MLLMs underperform in safety-critical scenarios compared to normal scenes, but fine-tuning with WaymoQA significantly improves their reasoning ability, highlighting the effectiveness of our dataset in developing safer and more reasoning-capable driving agents.

Related papers

• Advancing Adaptive Multi-Stage Video Anomaly Reasoning: A Benchmark Dataset and Method [96.63801368613177]
  We present a new task that elevates video anomaly analysis from descriptive understanding to structured, multi-stage reasoning.<n>We present a new dataset with 8,641 videos, totaling more than 50,000 samples, making it one of the largest datasets for video anomaly understanding.<n>Building upon the proposed task and dataset, we develop an end-to-end MLLM-based VAR model termed Vad-R1-Plus, which supports adaptive hierarchical reasoning and risk-aware decision making.
  arXiv  Detail & Related papers  (2026-01-15T08:09:04Z)
• Risk-adaptive Activation Steering for Safe Multimodal Large Language Models [25.347491265330863]
  One of the key challenges of modern AI models is ensuring they provide helpful responses to benign queries while refusing malicious ones.<n>We propose to reformulate queries to strengthen cross-modal attention to safety-critical image regions.<n>Using the assessed risk, it adaptively steers activations to generate responses that are safe and helpful without overhead from iterative output adjustments.
  arXiv  Detail & Related papers  (2025-10-15T15:57:17Z)
• Automating Steering for Safe Multimodal Large Language Models [58.36932318051907]
  We introduce a modular and adaptive inference-time intervention technology, AutoSteer, without requiring any fine-tuning of the underlying model.<n>AutoSteer incorporates three core components: (1) a novel Safety Awareness Score (SAS) that automatically identifies the most safety-relevant distinctions among the model's internal layers; (2) an adaptive safety prober trained to estimate the likelihood of toxic outputs from intermediate representations; and (3) a lightweight Refusal Head that selectively intervenes to modulate generation when safety risks are detected.
  arXiv  Detail & Related papers  (2025-07-17T16:04:55Z)
• Multimodal Large Language Models for Enhanced Traffic Safety: A Comprehensive Review and Future Trends [5.233512464561313]
  Traffic safety remains a critical global challenge, with traditional Advanced Driver-Assistance Systems often struggling in dynamic real-world scenarios.<n>This paper reviews the transformative potential of Multimodal Large Language Models (MLLMs) in addressing these limitations.<n>By positioning MLLMs as a cornerstone for next-generation traffic safety systems, this review underscores their potential to revolutionize the field.
  arXiv  Detail & Related papers  (2025-04-21T18:48:35Z)
• DriveLMM-o1: A Step-by-Step Reasoning Dataset and Large Multimodal Model for Driving Scenario Understanding [76.3876070043663]
  We propose DriveLMM-o1, a dataset and benchmark designed to advance step-wise visual reasoning for autonomous driving.<n>Our benchmark features over 18k VQA examples in the training set and more than 4k in the test set, covering diverse questions on perception, prediction, and planning.<n>Our model achieves a +7.49% gain in final answer accuracy, along with a 3.62% improvement in reasoning score over the previous best open-source model.
  arXiv  Detail & Related papers  (2025-03-13T17:59:01Z)
• SafeAuto: Knowledge-Enhanced Safe Autonomous Driving with Multimodal Foundation Models [63.71984266104757]
  We propose SafeAuto, a framework that enhances MLLM-based autonomous driving by incorporating both unstructured and structured knowledge.<n>To explicitly integrate safety knowledge, we develop a reasoning component that translates traffic rules into first-order logic.<n>Our Multimodal Retrieval-Augmented Generation model leverages video, control signals, and environmental attributes to learn from past driving experiences.
  arXiv  Detail & Related papers  (2025-02-28T21:53:47Z)
• Black-Box Adversarial Attack on Vision Language Models for Autonomous Driving [65.61999354218628]
  We take the first step toward designing black-box adversarial attacks specifically targeting vision-language models (VLMs) in autonomous driving systems.<n>We propose Cascading Adversarial Disruption (CAD), which targets low-level reasoning breakdown by generating and injecting semantics.<n>We present Risky Scene Induction, which addresses dynamic adaptation by leveraging a surrogate VLM to understand and construct high-level risky scenarios.
  arXiv  Detail & Related papers  (2025-01-23T11:10:02Z)
• Are VLMs Ready for Autonomous Driving? An Empirical Study from the Reliability, Data, and Metric Perspectives [56.528835143531694]
  We introduce DriveBench, a benchmark dataset designed to evaluate Vision-Language Models (VLMs)<n>Our findings reveal that VLMs often generate plausible responses derived from general knowledge or textual cues rather than true visual grounding.<n>We propose refined evaluation metrics that prioritize robust visual grounding and multi-modal understanding.
  arXiv  Detail & Related papers  (2025-01-07T18:59:55Z)
• Is it safe to cross? Interpretable Risk Assessment with GPT-4V for Safety-Aware Street Crossing [8.468153670795443]
  This paper introduces an innovative approach that leverages large multimodal models (LMMs) to interpret complex street crossing scenes. By generating a safety score and scene description in natural language, our method supports safe decision-making for the blind and low-vision individuals.
  arXiv  Detail & Related papers  (2024-02-09T21:37:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.