MS-rPPG: Multi-spectral State Space Model for Remote Photoplethysmography in Driver Monitoring Systems

• URL: http://arxiv.org/abs/2606.21115v1
• Date: Fri, 19 Jun 2026 05:34:10 GMT
• Title: MS-rPPG: Multi-spectral State Space Model for Remote Photoplethysmography in Driver Monitoring Systems
• Abstract summary: In this study, we investigate a driver health monitoring system based on remote heart rate estimation.<n>We introduce MS-r, a multi-spectral framework that combines RGB with near-infrared (NIR) face video to alleviate r estimation under challenging driving conditions.<n>We collected a real-world dataset called MS-Drive, which was recorded from 50 participants while driving the vehicle.
• Score: 21.37687518779808
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Remote photoplethysmography (rPPG) is a camera-based technique for measuring physiological signals, particularly cardiac activity. From the remotely measured signals, heart rate can be estimated, which is crucial for health monitoring. In this study, we investigate a driver health monitoring system based on remote heart rate estimation. However, driving environments represent uncontrolled settings where videos are subject to varying illumination conditions and frequent head movements. We introduce MS-rPPG, a multi-spectral framework that combines RGB with near-infrared (NIR) face video to alleviate rPPG estimation under challenging driving conditions. To combine the complementary features from two spectral videos, we propose a cross-spectral linear modulation (CSLM) strategy based on frequency-domain analysis. Moreover, we introduce MS-Mamba, a novel state space model designed to effectively model long-range temporal dependencies while jointly capturing cross-channel interactions between multi-spectral features. We collected a real-world dataset called MS-Drive, which was recorded from 50 participants while driving the vehicle. The proposed method was evaluated on the MR-NIRP Car dataset and MS-Drive datasets. The experimental results indicate that MS-rPPG shows better robustness and heart rate estimation accuracy than previous methods, highlighting its promise for driver health monitoring. The codes are available at github.com/ziiho08/MS-rPPG.

Related papers

• MVRD-Bench: Multi-View Learning and Benchmarking for Dynamic Remote Photoplethysmography under Occlusion [38.964013784587436]
  Remote photoplethysmography (CFA) is a non-contactr technique that estimates physiological signals by analyzing subtle skin color changes in facial videos.<n>Existing r methods often encounter performance degradation under facial motion and scenarios due to their reliance on static and single-view facial videos.<n>This work focuses on tackling the motion-induced rhythmic occlusion problem for r measurement in un multi-view facial videos.
  arXiv  Detail & Related papers  (2026-03-24T06:01:56Z)
• PhysNeXt: Next-Generation Dual-Branch Structured Attention Fusion Network for Remote Photoplethysmography Measurement [50.524262997433546]
  rmography enables measurement of heart rate and other vital signs by analyzing color variations in facial skin induced by cardiac pulsation.<n>Current methods are mainly based on either end-to-end modeling from raw videos or subtle heartbeat-related map (ST) representations.<n>We propose PhysMapXt, a dual-input deep learning framework that jointly exploits video frames and ST representations.
  arXiv  Detail & Related papers  (2026-03-20T08:37:02Z)
• EMPD: An Event-based Multimodal Physiological Dataset for Remote Pulse Wave Detection [17.621871238954053]
  EMPD is the first benchmark dataset specifically for non-contact physiological sensing via event cameras.<n>It provides synchronized precisely multimodal data with microsecond-level temporal precision.<n>The dataset serves as a crucial resource for developing robust algorithms in the field of neuromorphic physiological monitoring.
  arXiv  Detail & Related papers  (2026-03-17T01:25:05Z)
• Reperio-rPPG: Relational Temporal Graph Neural Networks for Periodicity Learning in Remote Physiological Measurement [0.0]
  Remote photoplethys (rmography) is an emerging physiological sensing technique that leverages subtle color variations in facial videos to estimate vital signs such as heart rate and respiratory rate.<n>This non-invasive technique has gained traction across diverse domains, but its ability to capture fine-grained temporal dynamics under real-world conditions has been underexplored.<n>We propose Graph Reperio-r, a novel framework that strategically integrates a Transformer to effectively capture the periodic structure.
  arXiv  Detail & Related papers  (2025-11-08T09:41:34Z)
• TYrPPG: Uncomplicated and Enhanced Learning Capability rPPG for Remote Heart Rate Estimation [51.56484100374058]
  This paper introduces an innovative video understanding block (GVB) designed for efficient RGB videos.<n>Based on the Mam structure, this block integrates 2D-CNN and 3D-CNN to enhance video understanding for analysis.<n>Experiments show that our TYr can achieve state-of-the-art performance in commonly used datasets.
  arXiv  Detail & Related papers  (2025-11-08T03:46:58Z)
• PHASE-Net: Physics-Grounded Harmonic Attention System for Efficient Remote Photoplethysmography Measurement [63.007237197267834]
  Existing deep learning methods are mostly physiological monitoring and lack theoretical robustness.<n>We propose a physics-informed r paradigm derived from the Navier-Stokes equations of hemodynamics, showing that the pulse signal follows a second-order system.<n>This provides a theoretical justification for using a Temporal Conal Network (TCN)<n>Phase-Net achieves state-of-the-art performance with strong efficiency, offering a theoretically grounded and deployment-ready r solution.
  arXiv  Detail & Related papers  (2025-09-29T14:36:45Z)
• Tracking the Unstable: Appearance-Guided Motion Modeling for Robust Multi-Object Tracking in UAV-Captured Videos [58.156141601478794]
  Multi-object tracking (UAVT) aims to track multiple objects while maintaining consistent identities across frames of a given video.<n>Existing methods typically model motion cues and appearance separately, overlooking their interplay and resulting in suboptimal tracking performance.<n>We propose AMOT, which exploits appearance and motion cues through two key components: an Appearance-Motion Consistency (AMC) matrix and a Motion-aware Track Continuation (MTC) module.
  arXiv  Detail & Related papers  (2025-08-03T12:06:47Z)
• Time-Series U-Net with Recurrence for Noise-Robust Imaging Photoplethysmography [14.749406169315554]
  Photoplethysmography system consists of three modules: face and landmark detection, time-series extraction, and pulse signal/pulse rate estimation.<n>The pulse signal estimation module, which we call TURNIP, allows the system to faithfully reconstruct the underlying pulse signal waveform.<n>Our algorithm provides reliable heart rate estimates without the need for specialized sensors or contact with the skin.
  arXiv  Detail & Related papers  (2025-03-21T17:52:33Z)
• Leveraging Driver Field-of-View for Multimodal Ego-Trajectory Prediction [69.29802752614677]
  RouteFormer is a novel ego-trajectory prediction network combining GPS data, environmental context, and the driver's field-of-view.<n>To tackle data scarcity and enhance diversity, we introduce GEM, a dataset of urban driving scenarios enriched with synchronized driver field-of-view and gaze data.
  arXiv  Detail & Related papers  (2023-12-13T23:06:30Z)
• Facial Video-based Remote Physiological Measurement via Self-supervised Learning [9.99375728024877]
  We introduce a novel framework that learns to estimate r signals from facial videos without the need of ground truth signals. Negative samples are generated via a learnable frequency module, which performs nonlinear signal frequency transformation. Next, we introduce a local r expert aggregation module to estimate r signals from augmented samples. It encodes complementary pulsation information from different face regions and aggregate them into one r prediction.
  arXiv  Detail & Related papers  (2022-10-27T13:03:23Z)
• PhysFormer: Facial Video-based Physiological Measurement with Temporal Difference Transformer [55.936527926778695]
  Recent deep learning approaches focus on mining subtle r clues using convolutional neural networks with limited-temporal receptive fields. In this paper, we propose the PhysFormer, an end-to-end video transformer based architecture.
  arXiv  Detail & Related papers  (2021-11-23T18:57:11Z)
• Video-based Remote Physiological Measurement via Cross-verified Feature Disentangling [121.50704279659253]
  We propose a cross-verified feature disentangling strategy to disentangle the physiological features with non-physiological representations. We then use the distilled physiological features for robust multi-task physiological measurements. The disentangled features are finally used for the joint prediction of multiple physiological signals like average HR values and r signals.
  arXiv  Detail & Related papers  (2020-07-16T09:39:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.