Risk-Aware Selective Multimodal Driver Monitoring with Driver-State World Modeling

• URL: http://arxiv.org/abs/2606.26922v1
• Date: Thu, 25 Jun 2026 11:59:21 GMT
• Title: Risk-Aware Selective Multimodal Driver Monitoring with Driver-State World Modeling
• Abstract summary: Continuous driver monitoring in automated vehicles requires low-latency inference.<n>We propose a cost-aware selective inference framework for deployable multimodal driver monitoring.
• Score: 13.540675152706589
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Continuous driver monitoring in automated vehicles requires low-latency inference while avoiding unsafe decisions under uncertain driver states. Large vision-language models provide broad multimodal priors, but their latency and limited reliability in this setting make them unsuitable as always-on in-cabin monitors. We propose a cost-aware selective inference framework for deployable multimodal driver monitoring. The core system is a lightweight RGB-physiological student that combines in-cabin visual observations with window-level HR/EDA signals, and a learned gate that decides when to accept the fast prediction or abstain for safety intervention. Additional controls show that the learned scores contain sample-level information beyond scenario priors, while exact physiological synchronization remains a limitation. To incorporate predictive evidence, we further study a compact driver-state world modeling module that rolls out latent driver-state features and estimates future fast-model errors and counterfactual system-level action costs. On scenario-induced driver-demand recognition, the RGB-physiological student improves over RGB-only and physiology-only baselines, reaching 0.7440 Macro-F1 and 0.9099 balanced accuracy with 11.39M parameters and 3.08ms inference latency. Cost-aware selective inference reduces unsafe false negatives from 17.37% under always-fast inference to approximately 5% across seeds, while maintaining deployment-level latency. While driver-state world modeling offers valuable predictive signals, worst-group evaluations highlight persistent operating-point calibration drift. Ultimately, reliable edge driver monitoring requires advancing not only perception backbones, but also risk-aware selective control and group-robust calibration.

Related papers

• A Neural Network Based Teleoperation for Remote Controlled Vehicles [98.38733381483934]
  Direct teleoperation of vehicles faces critical technical bottlenecks.<n> communication latency and the operator's inability to physically perceive unmodeled environmental disturbances.<n>We propose a tailored unilateral teleoperation framework.
  arXiv  Detail & Related papers  (2026-08-11T01:51:08Z)
• Taming Perception Jitter: Uncertainty-Aware LiDAR Object Detection for Reliable Motion Classification [8.046180274721541]
  Unstable bounding box predictions can lead to spurious velocity estimates in tracking and falsely predicted trajectories.<n>We present a deployment-friendly mitigation strategy that augments a 3D object detector with aleatoric uncertainty estimates and applies a two-sample z-test over short observation windows to separate true motion from jitter.
  arXiv  Detail & Related papers  (2026-06-08T11:24:57Z)
• Bench2Drive-Robust: Benchmarking Closed-Loop Autonomous Driving under Deployment Perturbations [122.22391796628408]
  We present Bench2Drive-Robust, to our knowledge the first device-centric robustness benchmark for closed-loop end-to-end autonomous driving.<n>We systematically evaluate deployment-oriented perturbations arising from three major sources: camera-stream failures, ego-state estimation errors, and compute-induced control delay.<n>Our results show that these deployment-related perturbations can substantially degrade closed-loop driving performance.
  arXiv  Detail & Related papers  (2026-05-18T08:45:24Z)
• Driver-WM: A Driver-Centric Traffic-Conditioned Latent World Model for In-Cabin Dynamics Rollout [55.19244743715421]
  Driver-WM is a driver-centric latent world model that rolls out in-cabin dynamics causally conditioned on out-cabin traffic context.<n>It unifies physical kinematics forecasting with auxiliary behavioral and emotional semantic recognition.<n>Driver-WM yields robust long-horizon geometric forecasting for reactive high-motion maneuvers.
  arXiv  Detail & Related papers  (2026-05-06T16:30:48Z)
• Real-Time In-Cabin Driver Behavior Recognition on Low-Cost Edge Hardware [1.1100764382749708]
  Single-camera in-cabin driver behavior recognition system designed for deployment on Raspberry Pi 5 and Google Coral Edge TPU.<n>System covers 17 behavior classes, including multiple phone-use modes, eating/drinking, smoking, reaching behind, gaze/attention shifts, passenger interaction, grooming, control-panel interaction, yawning, and eyes-closed sleep.
  arXiv  Detail & Related papers  (2025-12-26T00:54:24Z)
• Optimization-Guided Diffusion for Interactive Scene Generation [52.23368750264419]
  We present OMEGA, an optimization-guided, training-free framework that enforces structural consistency and interaction awareness during diffusion-based sampling.<n>We show that OMEGA improves generation realism, consistency, and controllability, increasing the ratio of physically and behaviorally valid scenes.<n>Our approach can also generate $5times$ more near-collision frames with a time-to-collision under three seconds.
  arXiv  Detail & Related papers  (2025-12-08T15:56:18Z)
• How Much Is Too Much? Adaptive, Context-Aware Risk Detection in Naturalistic Driving [0.6299766708197883]
  We propose a unified, context-aware framework that adapts labels and models over time and across drivers.<n>The framework is tested using two safety indicators, speed-weighted headway and harsh driving events, and three models: Random Forest, XGBoost, and Deep Neural Network (DNN)<n>Overall, the framework shows promise for adaptive, context-aware risk detection that can enhance real-time safety feedback and support driver-focused interventions in intelligent transportation systems.
  arXiv  Detail & Related papers  (2025-07-26T16:24:25Z)
• Driver-Net: Multi-Camera Fusion for Assessing Driver Take-Over Readiness in Automated Vehicles [5.543220407902113]
  Driver-Net is a novel deep learning framework that fuses multi-camera inputs to estimate driver take-over readiness.<n>It captures synchronised visual cues from the driver's head, hands, and body posture through a triple-camera setup.<n>The proposed method achieves an accuracy of up to 95.8% in driver readiness classification.
  arXiv  Detail & Related papers  (2025-07-05T19:27:03Z)
• SafeAuto: Knowledge-Enhanced Safe Autonomous Driving with Multimodal Foundation Models [63.71984266104757]
  We propose SafeAuto, a framework that enhances MLLM-based autonomous driving by incorporating both unstructured and structured knowledge.<n>To explicitly integrate safety knowledge, we develop a reasoning component that translates traffic rules into first-order logic.<n>Our Multimodal Retrieval-Augmented Generation model leverages video, control signals, and environmental attributes to learn from past driving experiences.
  arXiv  Detail & Related papers  (2025-02-28T21:53:47Z)
• Cross-Camera Distracted Driver Classification through Feature Disentanglement and Contrastive Learning [13.613407983544427]
  Driver Behavior Monitoring Network (DBMNet) relies on a lightweight backbone and integrates a disentanglement module to discard camera view information.<n>DBMNet achieves an improvement of 7% in Top-1 accuracy compared to existing approaches.
  arXiv  Detail & Related papers  (2024-11-20T10:27:12Z)
• RACER: Rational Artificial Intelligence Car-following-model Enhanced by Reality [46.909086734963665]
  This paper introduces RACER, a cutting-edge deep learning car-following model to predict Adaptive Cruise Control (ACC) driving behavior.<n>Unlike conventional models, RACER effectively integrates Rational Driving Constraints (RDCs), crucial tenets of actual driving.<n> RACER excels across key metrics, such as acceleration, velocity, and spacing, registering zero violations.
  arXiv  Detail & Related papers  (2023-12-12T06:21:30Z)
• Efficient and Robust LiDAR-Based End-to-End Navigation [132.52661670308606]
  We present an efficient and robust LiDAR-based end-to-end navigation framework. We propose Fast-LiDARNet that is based on sparse convolution kernel optimization and hardware-aware model design. We then propose Hybrid Evidential Fusion that directly estimates the uncertainty of the prediction from only a single forward pass.
  arXiv  Detail & Related papers  (2021-05-20T17:52:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.