InCarEmo: A Multimodal Dataset for In-Cabin Emotion Recognition and Driver State Monitoring

• URL: http://arxiv.org/abs/2607.14683v1
• Date: Thu, 16 Jul 2026 07:44:42 GMT
• Title: InCarEmo: A Multimodal Dataset for In-Cabin Emotion Recognition and Driver State Monitoring
• Abstract summary: InCarEmo is a multimodal dataset for in-cabin emotion recognition and driver state monitoring.<n>It integrates RGB and infrared video, in-cabin audio, and dialogue text collected from scripted in-cabin scenarios.<n>We provide a unified benchmark with extensive baseline results across unimodal and multimodal methods.
• Score: 41.33240617747301
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Understanding driver emotion and state is critical for the next generation of intelligent in-cabin systems that ensure safety and enhance human-vehicle interaction. However, existing public datasets for in-cabin affective computing are largely limited to visual modalities and rarely include conversational information, making it difficult to capture the linguistic and interactive cues underlying driver emotion. To address these gaps, we introduce InCarEmo, a multimodal dataset for in-cabin emotion recognition and driver state monitoring. InCarEmo integrates RGB and infrared video, in-cabin audio, and dialogue text collected from scripted in-cabin scenarios designed to simulate realistic driver behaviors, covering diverse lighting conditions and driving contexts. The dataset supports three primary tasks: 1) multimodal emotion recognition, 2) fatigue detection, and 3) distraction monitoring. In addition to the original Chinese data, we construct an auxiliary English benchmark to support preliminary cross-lingual evaluation. We provide a unified benchmark with extensive baseline results across unimodal and multimodal methods, including analyses under modality-missing and noise conditions. Experimental results demonstrate the benefits of multimodal fusion and reveal remaining challenges under real-world noise and low-light conditions. By releasing InCarEmo, we aim to establish a comprehensive foundation for robust, interpretable, and human-centric in-cabin affective understanding, promoting safer and more empathetic driver-vehicle interaction.

Related papers

• NARRATE: A Multimodal Real-World Australian Driving Dataset for Human-Centred Explanations in Automated Driving [14.607239446674479]
  Existing language-annotated driving datasets are mostly observer-written, post-hoc, simulation-based, or generated from sensor inputs.<n>We introduce NARRATE, a multimodal real-world Australian driving dataset comprising 2,050 annotated events from 35 experienced drivers and driving instructors on public roads.<n>Each event is grounded in synchronised visual, localisation, motion, and LiDAR streams and paired with in-vehicle and/or post-drive free-text explanations.
  arXiv  Detail & Related papers  (2026-08-14T12:35:24Z)
• Tactile-based Multimodal Fusion in Embodied Intelligence: A Survey of Vision, Language, and Contact-Driven Paradigms [70.51538670020267]
  We propose a hierarchical taxonomy that organizes the field into two primary dimensions: multimodal datasets and multimodal methods.<n>On the data side, we categorize resources ranging from Tactile-Vision datasets, Tactile-Language datasets, Tactile-Vision-Language datasets, and Tactile-Vision-Other datasets.<n>On the method side, we structure prior work into three core pillars: (1) Multimodal Perception and Recognition, (2) Cross-Modal Generation, focusing on bidirectional translation between tactile, vision, and text, and (3) Multimodal Interaction, emphasizing feedback control and language-guided manipulation.
  arXiv  Detail & Related papers  (2026-05-17T09:09:30Z)
• Looking and Listening Inside and Outside: Multimodal Artificial Intelligence Systems for Driver Safety Assessment and Intelligent Vehicle Decision-Making [30.720516012789357]
  This research makes a case for the audio modality as an additional source of information to understand the driver.<n>We expand LILO by incorporating audio signals, forming the looking-and-listening inside-and-outside (L-LIO) framework.<n>Pilot findings show that audio yields safety-relevant insights, particularly in nuanced or context-rich scenarios.
  arXiv  Detail & Related papers  (2026-02-07T19:25:02Z)
• Multimodal Large Language Model Framework for Safe and Interpretable Grid-Integrated EVs [3.7098231493739764]
  This paper presents a multi-modal large language model (LLM)-based framework to process multimodal sensor data.<n>The framework is validated using real-world data collected from instrumented vehicles driving on urban roads.<n>By combining visual perception (YOLOv8), geocoded positioning, and CAN bus telemetry, the framework bridges raw sensor data and driver comprehension.
  arXiv  Detail & Related papers  (2025-10-02T21:50:31Z)
• Efficient Mixture-of-Expert for Video-based Driver State and Physiological Multi-task Estimation in Conditional Autonomous Driving [12.765198683804094]
  Road safety remains a critical challenge worldwide, with approximately 1.35 million fatalities annually attributed to traffic accidents.<n>We propose a novel multi-task DMS, termed VDMoE, which leverages RGB video input to monitor driver states non-invasively.
  arXiv  Detail & Related papers  (2024-10-28T14:49:18Z)
• Probing Multimodal LLMs as World Models for Driving [72.18727651074563]
  We look at the application of Multimodal Large Language Models (MLLMs) in autonomous driving. Despite advances in models like GPT-4o, their performance in complex driving environments remains largely unexplored.
  arXiv  Detail & Related papers  (2024-05-09T17:52:42Z)
• Leveraging Driver Field-of-View for Multimodal Ego-Trajectory Prediction [69.29802752614677]
  RouteFormer is a novel ego-trajectory prediction network combining GPS data, environmental context, and the driver's field-of-view.<n>To tackle data scarcity and enhance diversity, we introduce GEM, a dataset of urban driving scenarios enriched with synchronized driver field-of-view and gaze data.
  arXiv  Detail & Related papers  (2023-12-13T23:06:30Z)
• Drive Anywhere: Generalizable End-to-end Autonomous Driving with Multi-modal Foundation Models [114.69732301904419]
  We present an approach to apply end-to-end open-set (any environment/scene) autonomous driving that is capable of providing driving decisions from representations queryable by image and text. Our approach demonstrates unparalleled results in diverse tests while achieving significantly greater robustness in out-of-distribution situations.
  arXiv  Detail & Related papers  (2023-10-26T17:56:35Z)
• AIDE: A Vision-Driven Multi-View, Multi-Modal, Multi-Tasking Dataset for Assistive Driving Perception [26.84439405241999]
  We present an AssIstive Driving pErception dataset (AIDE) that considers context information both inside and outside the vehicle. AIDE facilitates holistic driver monitoring through three distinctive characteristics. Two fusion strategies are introduced to give new insights into learning effective multi-stream/modal representations.
  arXiv  Detail & Related papers  (2023-07-26T03:12:05Z)
• Exploring Contextual Representation and Multi-Modality for End-to-End Autonomous Driving [58.879758550901364]
  Recent perception systems enhance spatial understanding with sensor fusion but often lack full environmental context. We introduce a framework that integrates three cameras to emulate the human field of view, coupled with top-down bird-eye-view semantic data to enhance contextual representation. Our method achieves displacement error by 0.67m in open-loop settings, surpassing current methods by 6.9% on the nuScenes dataset.
  arXiv  Detail & Related papers  (2022-10-13T05:56:20Z)
• Fine-Grained Vehicle Perception via 3D Part-Guided Visual Data Augmentation [77.60050239225086]
  We propose an effective training data generation process by fitting a 3D car model with dynamic parts to vehicles in real images. Our approach is fully automatic without any human interaction. We present a multi-task network for VUS parsing and a multi-stream network for VHI parsing.
  arXiv  Detail & Related papers  (2020-12-15T03:03:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.