DriveXQA: Cross-modal Visual Question Answering for Adverse Driving Scene Understanding

• URL: http://arxiv.org/abs/2603.11380v1
• Date: Wed, 11 Mar 2026 23:49:11 GMT
• Title: DriveXQA: Cross-modal Visual Question Answering for Adverse Driving Scene Understanding
• Abstract summary: We propose the DriveXQA, a multimodal dataset for autonomous driving VQA.<n>In addition to four visual modalities, five sensor failure cases, and five weather conditions, it includes $102,505$ QA pairs categorized into three types: global scene level, allocentric level, and ego-vehicle centric level.<n>We design MVX-LLM, a token-efficient architecture with a Dual Cross-Attention (DCA) projector that fuses the modalities to alleviate information redundancy.
• Score: 40.93547456625219
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Fusing sensors with complementary modalities is crucial for maintaining a stable and comprehensive understanding of abnormal driving scenes. However, Multimodal Large Language Models (MLLMs) are underexplored for leveraging multi-sensor information to understand adverse driving scenarios in autonomous vehicles. To address this gap, we propose the DriveXQA, a multimodal dataset for autonomous driving VQA. In addition to four visual modalities, five sensor failure cases, and five weather conditions, it includes $102,505$ QA pairs categorized into three types: global scene level, allocentric level, and ego-vehicle centric level. Since no existing MLLM framework adopts multiple complementary visual modalities as input, we design MVX-LLM, a token-efficient architecture with a Dual Cross-Attention (DCA) projector that fuses the modalities to alleviate information redundancy. Experiments demonstrate that our DCA achieves improved performance under challenging conditions such as foggy (GPTScore: $53.5$ vs. $25.1$ for the baseline). The established dataset and source code will be made publicly available.

Related papers

• V2X-QA: A Comprehensive Reasoning Dataset and Benchmark for Multimodal Large Language Models in Autonomous Driving Across Ego, Infrastructure, and Cooperative Views [22.24590004859344]
  V2X-QA is a real-world dataset and benchmark for evaluating MLLMs across vehicle-side, infrastructure-side, and cooperative viewpoints.<n>Results show viewpoint accessibility substantially affects performance, and infrastructure-side reasoning supports meaningful macroscopic traffic understanding.<n>We introduce V2X-MoE, a benchmark-aligned baseline with explicit view routing and viewpoint-specific LoRA experts.
  arXiv  Detail & Related papers  (2026-04-03T04:07:35Z)
• Bench2Drive-VL: Benchmarks for Closed-Loop Autonomous Driving with Vision-Language Models [50.22099309218635]
  In autonomous driving, closed-loop evaluation has become widely recognized as a more reliable validation method than open-loop evaluation.<n>We present Bench2Drive-VL, an extension of Bench2Drive that brings closed-loop evaluation to VLM-based driving.
  arXiv  Detail & Related papers  (2026-04-01T11:38:46Z)
• UAV traffic scene understanding: A regulation embedded multi-modal network and a unified benchmark [22.241798167957214]
  Traffic scene understanding from unmanned aerial vehicle (UAV) platforms is crucial for intelligent transportation systems.<n>Existing methods face significant challenges in real-world surveillance, as their heavy reliance on optical imagery leads to severe performance degradation.<n>We propose a novel Multi-modal Traffic Cognition Network (MTCNet) for robust UAV traffic scene understanding.
  arXiv  Detail & Related papers  (2026-03-11T12:53:38Z)
• MMDrive: Interactive Scene Understanding Beyond Vision with Multi-representational Fusion [39.303609347179695]
  This study proposes MMDrive, a vision-language model framework that extends traditional image understanding to a generalized 3D scene understanding framework.<n> MMDrive incorporates three complementary modalities, including occupancy maps, LiDAR point clouds, and textual scene descriptions.<n> MMDrive achieves significant performance gains over existing vision-language models for autonomous driving, with a BLEU-4 score of 54.56 and METEOR of 41.78 on DriveLM, and an accuracy score of 62.7% on NuScenes-QA.
  arXiv  Detail & Related papers  (2025-12-15T10:37:59Z)
• RoboDriveVLM: A Novel Benchmark and Baseline towards Robust Vision-Language Models for Autonomous Driving [45.00491493068162]
  Current Vision-Language Model (VLM)-based end-to-end autonomous driving systems often leverage large language models to generate driving decisions.<n>This benchmark systematically evaluates two critical categories of real-world challenges for VLM-based end-to-end autonomous driving systems.<n>We propose a novel VLM-based autonomous driving framework called RoboDriveVLM, which enhances robustness by mapping more multimodal data-e.g., lidar and radar-into a unified latent space.
  arXiv  Detail & Related papers  (2025-12-01T05:44:06Z)
• A Tri-Modal Dataset and a Baseline System for Tracking Unmanned Aerial Vehicles [74.8162337823142]
  MM-UAV is the first large-scale benchmark for Multi-Modal UAV Tracking.<n>The dataset spans over 30 challenging scenarios, with 1,321 synchronised multi-modal sequences, and more than 2.8 million annotated frames.<n>Accompanying the dataset, we provide a novel multi-modal multi-UAV tracking framework.
  arXiv  Detail & Related papers  (2025-11-23T08:42:17Z)
• DriveLMM-o1: A Step-by-Step Reasoning Dataset and Large Multimodal Model for Driving Scenario Understanding [76.3876070043663]
  We propose DriveLMM-o1, a dataset and benchmark designed to advance step-wise visual reasoning for autonomous driving.<n>Our benchmark features over 18k VQA examples in the training set and more than 4k in the test set, covering diverse questions on perception, prediction, and planning.<n>Our model achieves a +7.49% gain in final answer accuracy, along with a 3.62% improvement in reasoning score over the previous best open-source model.
  arXiv  Detail & Related papers  (2025-03-13T17:59:01Z)
• Towards Intelligent Transportation with Pedestrians and Vehicles In-the-Loop: A Surveillance Video-Assisted Federated Digital Twin Framework [62.47416496137193]
  We propose a surveillance video assisted federated digital twin (SV-FDT) framework to empower ITSs with pedestrians and vehicles in-the-loop.<n>The architecture consists of three layers: (i) the end layer, which collects traffic surveillance videos from multiple sources; (ii) the edge layer, responsible for semantic segmentation-based visual understanding, twin agent-based interaction modeling, and local digital twin system (LDTS) creation in local regions; and (iii) the cloud layer, which integrates LDTSs across different regions to construct a global DT model in realtime.
  arXiv  Detail & Related papers  (2025-03-06T07:36:06Z)
• RoboSense: Large-scale Dataset and Benchmark for Egocentric Robot Perception and Navigation in Crowded and Unstructured Environments [62.5830455357187]
  We setup an egocentric multi-sensor data collection platform based on 3 main types of sensors (Camera, LiDAR and Fisheye)<n>A large-scale multimodal dataset is constructed, named RoboSense, to facilitate egocentric robot perception.
  arXiv  Detail & Related papers  (2024-08-28T03:17:40Z)
• Multi-Frame, Lightweight & Efficient Vision-Language Models for Question Answering in Autonomous Driving [0.0]
  We develop an efficient, lightweight, multi-frame vision language model which performs Visual Question Answering for autonomous driving. In comparison to previous approaches, EM-VLM4AD requires at least 10 times less memory and floating point operations.
  arXiv  Detail & Related papers  (2024-03-28T21:18:33Z)
• M2DA: Multi-Modal Fusion Transformer Incorporating Driver Attention for Autonomous Driving [11.36165122994834]
  We propose a Multi-Modal fusion transformer incorporating Driver Attention (M2DA) for autonomous driving. By incorporating driver attention, we empower the human-like scene understanding ability to autonomous vehicles to identify crucial areas precisely and ensure safety.
  arXiv  Detail & Related papers  (2024-03-19T08:54:52Z)
• V2V4Real: A Real-world Large-scale Dataset for Vehicle-to-Vehicle Cooperative Perception [49.7212681947463]
  Vehicle-to-Vehicle (V2V) cooperative perception system has great potential to revolutionize the autonomous driving industry. We present V2V4Real, the first large-scale real-world multi-modal dataset for V2V perception. Our dataset covers a driving area of 410 km, comprising 20K LiDAR frames, 40K RGB frames, 240K annotated 3D bounding boxes for 5 classes, and HDMaps.
  arXiv  Detail & Related papers  (2023-03-14T02:49:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.