BATON: A Multimodal Benchmark for Bidirectional Automation Transition Observation in Naturalistic Driving

• URL: http://arxiv.org/abs/2604.07263v1
• Date: Wed, 08 Apr 2026 16:29:24 GMT
• Title: BATON: A Multimodal Benchmark for Bidirectional Automation Transition Observation in Naturalistic Driving
• Abstract summary: Existing driving automation (DA) systems rely on human drivers to decide when to engage DA.<n>BATON is a naturalistic dataset capturing real-world DA usage across 127 drivers, and 136.6 hours of driving.<n>The dataset synchronizes front-view video, in-cabin video, decoded CAN bus signals, radar-based lead-vehicle interaction, and GPS-derived route context.<n>Results show that front-view video captures road context but not driver state, while in-cabin video reflects driver readiness but not the external scene.
• Score: 18.15118596168445
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Existing driving automation (DA) systems on production vehicles rely on human drivers to decide when to engage DA while requiring them to remain continuously attentive and ready to intervene. This design demands substantial situational judgment and imposes significant cognitive load, leading to steep learning curves, suboptimal user experience, and safety risks from both over-reliance and delayed takeover. Predicting when drivers hand over control to DA and when they take it back is therefore critical for designing proactive, context-aware HMI, yet existing datasets rarely capture the multimodal context, including road scene, driver state, vehicle dynamics, and route environment. To fill this gap, we introduce BATON, a large-scale naturalistic dataset capturing real-world DA usage across 127 drivers, and 136.6 hours of driving. The dataset synchronizes front-view video, in-cabin video, decoded CAN bus signals, radar-based lead-vehicle interaction, and GPS-derived route context, forming a closed-loop multimodal record around each control transition. We define three benchmark tasks: driving action understanding, handover prediction, and takeover prediction, and evaluate baselines spanning sequence models, classical classifiers, and zero-shot VLMs. Results show that visual input alone is insufficient for reliable transition prediction: front-view video captures road context but not driver state, while in-cabin video reflects driver readiness but not the external scene. Incorporating CAN and route-context signals substantially improves performance over video-only settings, indicating strong complementarity across modalities. We further find takeover events develop more gradually and benefit from longer prediction horizons, whereas handover events depend more on immediate contextual cues, revealing an asymmetry with direct implications for HMI design in assisted driving systems.

Related papers

• SIREN-Bench: Behavior-Driven Generation and Evaluation of Emergency-Vehicle Interactions [4.523530339704703]
  We present textbfSIREN, a behavior-driven SUMO--CARLA co-language platform for generating EMVsimulation--civilian interactions.<n>We instantiate the platform as textbfSIREN-v1, comprising seven parameterized interaction templates across emergency levels L1--L3 and three behavior families.<n>We demonstrate the benchmark through three representative tasks: 3D object detection, trajectory prediction, and vision-language risk understanding.
  arXiv  Detail & Related papers  (2026-08-25T05:41:34Z)
• A Dynamic Scene Interaction Reasoning Framework for Scene-level Lane-Change Intention and Trajectory Prediction of Multiple Interacting Vehicles [6.830580384536778]
  This study proposes a dynamic scene graph attention framework that predicts the lane-change intention and future trajectory of every relevant vehicle within a local traffic scene.<n> Experiments on the NGSIM I-80, NGSIM US-101, and highD datasets demonstrate consistent improvements over competing baselines.
  arXiv  Detail & Related papers  (2026-07-02T23:55:53Z)
• MindVLA-U1: VLA Beats VA with Unified Streaming Architecture for Autonomous Driving [54.57163800903507]
  We present MindVLA-U1, the first unified streaming VLA architecture for autonomous driving.<n>A unified VLM backbone produces AR language tokens and flow-matching continuous action trajectories in a single forward pass over one shared representation.<n>On the long-tail WOD-E2E benchmark, MindVLA-U1 surpasses experienced human drivers for the first time.
  arXiv  Detail & Related papers  (2026-05-12T18:09:42Z)
• Driver-WM: A Driver-Centric Traffic-Conditioned Latent World Model for In-Cabin Dynamics Rollout [55.19244743715421]
  Driver-WM is a driver-centric latent world model that rolls out in-cabin dynamics causally conditioned on out-cabin traffic context.<n>It unifies physical kinematics forecasting with auxiliary behavioral and emotional semantic recognition.<n>Driver-WM yields robust long-horizon geometric forecasting for reactive high-motion maneuvers.
  arXiv  Detail & Related papers  (2026-05-06T16:30:48Z)
• LMGenDrive: Bridging Multimodal Understanding and Generative World Modeling for End-to-End Driving [60.31765454895336]
  We present LMGenDrive, the first framework that combines multimodal understanding with generative world models for end-to-end closed-loop driving.<n>We propose a progressive three-stage training strategy, from vision pretraining to multi-step long-horizon driving, to improve stability and performance.
  arXiv  Detail & Related papers  (2026-04-09T19:13:14Z)
• FlowDrive: Energy Flow Field for End-to-End Autonomous Driving [50.89871153094958]
  FlowDrive is a novel framework that introduces physically interpretable energy-based flow fields to encode semantic priors and safety cues into the BEV space.<n> Experiments on the NAVSIM v2 benchmark demonstrate that FlowDrive achieves state-of-the-art performance with anS of 86.3, surpassing prior baselines in both safety and planning quality.
  arXiv  Detail & Related papers  (2025-09-17T13:51:33Z)
• ImagiDrive: A Unified Imagination-and-Planning Framework for Autonomous Driving [64.12414815634847]
  Vision-Language Models (VLMs) and Driving World Models (DWMs) have independently emerged as powerful recipes addressing different aspects of this challenge.<n>We propose ImagiDrive, a novel end-to-end autonomous driving framework that integrates a VLM-based driving agent with a DWM-based scene imaginer.
  arXiv  Detail & Related papers  (2025-08-15T12:06:55Z)
• SafeAuto: Knowledge-Enhanced Safe Autonomous Driving with Multimodal Foundation Models [63.71984266104757]
  We propose SafeAuto, a framework that enhances MLLM-based autonomous driving by incorporating both unstructured and structured knowledge.<n>To explicitly integrate safety knowledge, we develop a reasoning component that translates traffic rules into first-order logic.<n>Our Multimodal Retrieval-Augmented Generation model leverages video, control signals, and environmental attributes to learn from past driving experiences.
  arXiv  Detail & Related papers  (2025-02-28T21:53:47Z)
• Leveraging Driver Field-of-View for Multimodal Ego-Trajectory Prediction [69.29802752614677]
  RouteFormer is a novel ego-trajectory prediction network combining GPS data, environmental context, and the driver's field-of-view.<n>To tackle data scarcity and enhance diversity, we introduce GEM, a dataset of urban driving scenarios enriched with synchronized driver field-of-view and gaze data.
  arXiv  Detail & Related papers  (2023-12-13T23:06:30Z)
• CEMFormer: Learning to Predict Driver Intentions from In-Cabin and External Cameras via Spatial-Temporal Transformers [5.572431452586636]
  We introduce a new framework called Cross-View Episodic Memory Transformer (CEM) CEM employs unified memory representations to learn for an improved driver intention prediction. We propose a novel context-consistency loss that incorporates driving context as an auxiliary supervision signal to improve prediction performance.
  arXiv  Detail & Related papers  (2023-05-13T05:27:36Z)
• Context-Aware Timewise VAEs for Real-Time Vehicle Trajectory Prediction [4.640835690336652]
  We present ContextVAE, a context-aware approach for multi-modal vehicle trajectory prediction. Our approach takes into account both the social features exhibited by agents on the scene and the physical environment constraints. In all tested datasets, ContextVAE models are fast to train and provide high-quality multi-modal predictions in real-time.
  arXiv  Detail & Related papers  (2023-02-21T18:42:24Z)
• FBLNet: FeedBack Loop Network for Driver Attention Prediction [50.936478241688114]
  Nonobjective driving experience is difficult to model, so a mechanism simulating the driver experience accumulation procedure is absent in existing methods.<n>We propose a FeedBack Loop Network (FBLNet), which attempts to model the driving experience accumulation procedure.<n>Our model exhibits a solid advantage over existing methods, achieving an outstanding performance improvement on two driver attention benchmark datasets.
  arXiv  Detail & Related papers  (2022-12-05T08:25:09Z)
• Exploring Attention GAN for Vehicle Motion Prediction [2.887073662645855]
  We study the influence of attention in generative models for motion prediction, considering both physical and social context. We validate our method using the Argoverse Motion Forecasting Benchmark 1.1, achieving competitive unimodal results.
  arXiv  Detail & Related papers  (2022-09-26T13:18:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.