NARRATE: A Multimodal Real-World Australian Driving Dataset for Human-Centred Explanations in Automated Driving

• URL: http://arxiv.org/abs/2608.14767v1
• Date: Fri, 14 Aug 2026 12:35:24 GMT
• Title: NARRATE: A Multimodal Real-World Australian Driving Dataset for Human-Centred Explanations in Automated Driving
• Abstract summary: Existing language-annotated driving datasets are mostly observer-written, post-hoc, simulation-based, or generated from sensor inputs.<n>We introduce NARRATE, a multimodal real-world Australian driving dataset comprising 2,050 annotated events from 35 experienced drivers and driving instructors on public roads.<n>Each event is grounded in synchronised visual, localisation, motion, and LiDAR streams and paired with in-vehicle and/or post-drive free-text explanations.
• Score: 14.607239446674479
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Automated vehicles must explain their decisions in ways that passengers can understand, monitor, and trust. Existing language-annotated driving datasets are mostly observer-written, post-hoc, simulation-based, or generated from sensor inputs, rather than elicited from the driver performing the action. We introduce NARRATE, a multimodal real-world Australian driving dataset comprising 2,050 annotated events from 35 experienced drivers and driving instructors on public roads. Each event is grounded in synchronised visual, localisation, motion, and LiDAR streams and paired with in-vehicle and/or post-drive free-text explanations. NARRATE provides action labels, scenario-context labels spanning six high-level and 32 fine-grained categories, and span-level Situational Awareness (SA) annotations over driver explanations for Perception, Comprehension and Projection. Four benchmark tasks (SA, scenario-context, driver-action classification, and explanation generation) show that this structure is learnable from driver language, while fine-grained context recognition and explanation generation remain challenging. NARRATE paves a path towards more human-centred and domain-aware explanation models for automated driving.

Related papers

• The LAIA Dataset: Labelled Attention for Intelligent Automobiles [0.0]
  We present LAIA, a novel dataset designed to enrich end-to-end driving research with human attention data.<n>LAIA comprises over 15 hours of driving from 44 participants across carefully crafted scenarios designed to evoke natural responses.<n>We use LAIA to compare human attention with the perceptual attention emerging in our end-to-end driving models, thereby providing insight into their behavior.
  arXiv  Detail & Related papers  (2026-07-28T10:57:22Z)
• InCarEmo: A Multimodal Dataset for In-Cabin Emotion Recognition and Driver State Monitoring [41.33240617747301]
  InCarEmo is a multimodal dataset for in-cabin emotion recognition and driver state monitoring.<n>It integrates RGB and infrared video, in-cabin audio, and dialogue text collected from scripted in-cabin scenarios.<n>We provide a unified benchmark with extensive baseline results across unimodal and multimodal methods.
  arXiv  Detail & Related papers  (2026-07-16T07:44:42Z)
• Sce2DriveX: A Generalized MLLM Framework for Scene-to-Drive Learning [24.511628941825116]
  We introduce Sce2DriveX, a human-like driving chain-of-thought (CoT) reasoning framework framework.<n>It reconstructs the implicit cognitive chain inherent in human driving, covering scene understanding, meta-action reasoning, behavior interpretation analysis, motion planning and control.<n>It achieves state-of-the-art performance from scene understanding to end-to-end driving, as well as robust generalization on the CARLA Bench2Drive benchmark.
  arXiv  Detail & Related papers  (2025-02-19T09:50:44Z)
• HERMES: A Unified Self-Driving World Model for Simultaneous 3D Scene Understanding and Generation [59.675030933810106]
  We present a unified Driving World Model named HERMES.<n>We seamlessly integrate 3D scene understanding and future scene evolution (generation) through a unified framework in driving scenarios.<n> HERMES achieves state-of-the-art performance, reducing generation error by 32.4% and improving understanding metrics such as CIDEr by 8.0%.
  arXiv  Detail & Related papers  (2025-01-24T18:59:51Z)
• Leveraging Driver Field-of-View for Multimodal Ego-Trajectory Prediction [69.29802752614677]
  RouteFormer is a novel ego-trajectory prediction network combining GPS data, environmental context, and the driver's field-of-view.<n>To tackle data scarcity and enhance diversity, we introduce GEM, a dataset of urban driving scenarios enriched with synchronized driver field-of-view and gaze data.
  arXiv  Detail & Related papers  (2023-12-13T23:06:30Z)
• Assessing Drivers' Situation Awareness in Semi-Autonomous Vehicles: ASP based Characterisations of Driving Dynamics for Modelling Scene Interpretation and Projection [0.0]
  We present a framework to asses how aware the driver is about the situation and to provide human-centred assistance. The framework is developed as a modular system within the Robot Operating System (ROS) with modules for sensing the environment and the driver state. A particular focus of this paper is on an Answer Set Programming (ASP) based approach for modelling and reasoning about the driver's interpretation and projection of the scene.
  arXiv  Detail & Related papers  (2023-08-30T09:07:49Z)
• OpenLane-V2: A Topology Reasoning Benchmark for Unified 3D HD Mapping [84.65114565766596]
  We present OpenLane-V2, the first dataset on topology reasoning for traffic scene structure. OpenLane-V2 consists of 2,000 annotated road scenes that describe traffic elements and their correlation to the lanes. We evaluate various state-of-the-art methods, and present their quantitative and qualitative results on OpenLane-V2 to indicate future avenues for investigating topology reasoning in traffic scenes.
  arXiv  Detail & Related papers  (2023-04-20T16:31:22Z)
• DeepAccident: A Motion and Accident Prediction Benchmark for V2X Autonomous Driving [76.29141888408265]
  We propose a large-scale dataset containing diverse accident scenarios that frequently occur in real-world driving. The proposed DeepAccident dataset includes 57K annotated frames and 285K annotated samples, approximately 7 times more than the large-scale nuScenes dataset.
  arXiv  Detail & Related papers  (2023-04-03T17:37:00Z)
• Unsupervised Driving Event Discovery Based on Vehicle CAN-data [62.997667081978825]
  This work presents a simultaneous clustering and segmentation approach for vehicle CAN-data that identifies common driving events in an unsupervised manner. We evaluate our approach with a dataset of real Tesla Model 3 vehicle CAN-data and a two-hour driving session that we annotated with different driving events.
  arXiv  Detail & Related papers  (2023-01-12T13:10:47Z)
• Cognitive Accident Prediction in Driving Scenes: A Multimodality Benchmark [77.54411007883962]
  We propose a Cognitive Accident Prediction (CAP) method that explicitly leverages human-inspired cognition of text description on the visual observation and the driver attention to facilitate model training. CAP is formulated by an attentive text-to-vision shift fusion module, an attentive scene context transfer module, and the driver attention guided accident prediction module. We construct a new large-scale benchmark consisting of 11,727 in-the-wild accident videos with over 2.19 million frames.
  arXiv  Detail & Related papers  (2022-12-19T11:43:02Z)
• From Spoken Thoughts to Automated Driving Commentary: Predicting and Explaining Intelligent Vehicles' Actions [10.557942353553859]
  In commentary driving, drivers verbalise their observations, assessments and intentions. By speaking out their thoughts, both learning and expert drivers are able to create a better understanding and awareness of their surroundings. In the intelligent vehicle context, automated driving commentary can provide intelligible explanations about driving actions.
  arXiv  Detail & Related papers  (2022-04-19T19:39:13Z)
• Learning Accurate and Human-Like Driving using Semantic Maps and Attention [152.48143666881418]
  This paper investigates how end-to-end driving models can be improved to drive more accurately and human-like. We exploit semantic and visual maps from HERE Technologies and augment the existing Drive360 dataset with such. Our models are trained and evaluated on the Drive360 + HERE dataset, which features 60 hours and 3000 km of real-world driving data.
  arXiv  Detail & Related papers  (2020-07-10T22:25:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.