Your blush gives you away: detecting hidden mental states with remote photoplethysmography and thermal imaging

• URL: http://arxiv.org/abs/2401.09145v1
• Date: Wed, 17 Jan 2024 11:38:21 GMT
• Title: Your blush gives you away: detecting hidden mental states with remote photoplethysmography and thermal imaging
• Authors: Ivan Liu, Fangyuan Liu, Qi Zhong, Fei Ma, Shiguang Ni
• Abstract summary: This study aims to bridge the gap by integrating r-modal and thermal imaging to enhance prediction performance. Using r- alone, the prediction accuracy was 77% for cognitive stress and 61% for moral elevation, as determined by SVM. An early fusion strategy of these modalities significantly improved accuracies, achieving 87% for cognitive stress and 83% for moral elevation using RF.
• Score: 5.604613879541013
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multimodal emotion recognition techniques are increasingly essential for assessing mental states. Image-based methods, however, tend to focus predominantly on overt visual cues and often overlook subtler mental state changes. Psychophysiological research has demonstrated that HR and skin temperature are effective in detecting ANS activities, thereby revealing these subtle changes. However, traditional HR tools are generally more costly and less portable, while skin temperature analysis usually necessitates extensive manual processing. Advances in remote-PPG and automatic thermal ROI detection algorithms have been developed to address these issues, yet their accuracy in practical applications remains limited. This study aims to bridge this gap by integrating r-PPG with thermal imaging to enhance prediction performance. Ninety participants completed a 20-minute questionnaire to induce cognitive stress, followed by watching a film aimed at eliciting moral elevation. The results demonstrate that the combination of r-PPG and thermal imaging effectively detects emotional shifts. Using r-PPG alone, the prediction accuracy was 77% for cognitive stress and 61% for moral elevation, as determined by SVM. Thermal imaging alone achieved 79% accuracy for cognitive stress and 78% for moral elevation, utilizing a RF algorithm. An early fusion strategy of these modalities significantly improved accuracies, achieving 87% for cognitive stress and 83% for moral elevation using RF. Further analysis, which utilized statistical metrics and explainable machine learning methods including SHAP, highlighted key features and clarified the relationship between cardiac responses and facial temperature variations. Notably, it was observed that cardiovascular features derived from r-PPG models had a more pronounced influence in data fusion, despite thermal imaging's higher predictive accuracy in unimodal analysis.

Related papers

• Machine Learning Assisted Postural Movement Recognition using Photoplethysmography(PPG) [0.0]
  There is an urgent need for the development of fall detection and fall prevention technologies. This work presents for the first time the use of machine learning techniques to recognize postural movements. Various machine learning approaches were used for classification, and the Artificial Neural Network (ANN) was found to be the best.
  arXiv  Detail & Related papers  (2024-11-02T18:56:41Z)
• SympCam: Remote Optical Measurement of Sympathetic Arousal [41.97525156940624]
  We present a new convolutional architecture tailored to the task of remote sympathetic arousal prediction. We show that the sympathetic arousal predicted by our method allows detecting physical stress with a balanced accuracy of 90%. We contribute a dataset designed explicitly for the task of remote sympathetic arousal prediction.
  arXiv  Detail & Related papers  (2024-10-27T18:46:55Z)
• Analyzing Participants' Engagement during Online Meetings Using Unsupervised Remote Photoplethysmography with Behavioral Features [50.82725748981231]
  Engagement measurement finds application in healthcare, education, services. Use of physiological and behavioral features is viable, but impracticality of traditional physiological measurement arises due to the need for contact sensors. We demonstrate the feasibility of the unsupervised photoplethysmography (rmography) as an alternative for contact sensors.
  arXiv  Detail & Related papers  (2024-04-05T20:39:16Z)
• Wavelet Analysis of Noninvasive EEG Signals Discriminates Complex and Natural Grasp Types [0.16385815610837165]
  This research aims to decode hand grasps from Electroencephalograms (EEGs) for dexterous neuroprosthetic development and Brain-Computer Interface (BCI) applications.
  arXiv  Detail & Related papers  (2024-01-31T23:13:38Z)
• Distribution-based Low-rank Embedding [0.0]
  We propose James-Stein for eigenvector (JSE) and Weibull embedding approaches, as two novel strategies in computational thermography. The primary objective is to create a low-dimensional (LD) representation of the thermal data stream. The results of the proposed method indicate an enhancement in the projection of the predominant basis vector, yielding classification accuracy of 81.7%.
  arXiv  Detail & Related papers  (2023-12-29T12:29:45Z)
• Depression Recognition using Remote Photoplethysmography from Facial Videos [0.3867363075280544]
  Depression is a mental illness that may be harmful to an individual's health. This work analyzes physiological signals to observe if different depressive states have a noticeable impact on the blood volume pulse (BVP) and the heart rate variability (HRV) We propose a novel scheme that directly extracts them from facial videos, just based on visual information, removing the need for any contact-based device.
  arXiv  Detail & Related papers  (2022-06-09T10:23:49Z)
• Texture Characterization of Histopathologic Images Using Ecological Diversity Measures and Discrete Wavelet Transform [82.53597363161228]
  This paper proposes a method for characterizing texture across histopathologic images with a considerable success rate. It is possible to quantify the intrinsic properties of such images with promising accuracy on two HI datasets.
  arXiv  Detail & Related papers  (2022-02-27T02:19:09Z)
• Maximizing Self-supervision from Thermal Image for Effective Self-supervised Learning of Depth and Ego-motion [78.19156040783061]
  Self-supervised learning of depth and ego-motion from thermal images shows strong robustness and reliability under challenging scenarios. The inherent thermal image properties such as weak contrast, blurry edges, and noise hinder to generate effective self-supervision from thermal images. We propose an effective thermal image mapping method that significantly increases image information, such as overall structure, contrast, and details, while preserving temporal consistency.
  arXiv  Detail & Related papers  (2022-01-12T09:49:24Z)
• Video-based Remote Physiological Measurement via Cross-verified Feature Disentangling [121.50704279659253]
  We propose a cross-verified feature disentangling strategy to disentangle the physiological features with non-physiological representations. We then use the distilled physiological features for robust multi-task physiological measurements. The disentangled features are finally used for the joint prediction of multiple physiological signals like average HR values and r signals.
  arXiv  Detail & Related papers  (2020-07-16T09:39:17Z)
• Detecting Parkinsonian Tremor from IMU Data Collected In-The-Wild using Deep Multiple-Instance Learning [59.74684475991192]
  Parkinson's Disease (PD) is a slowly evolving neuro-logical disease that affects about 1% of the population above 60 years old. PD symptoms include tremor, rigidity and braykinesia. We present a method for automatically identifying tremorous episodes related to PD, based on IMU signals captured via a smartphone device.
  arXiv  Detail & Related papers  (2020-05-06T09:02:30Z)
• Continuous Emotion Recognition via Deep Convolutional Autoencoder and Support Vector Regressor [70.2226417364135]
  It is crucial that the machine should be able to recognize the emotional state of the user with high accuracy. Deep neural networks have been used with great success in recognizing emotions. We present a new model for continuous emotion recognition based on facial expression recognition.
  arXiv  Detail & Related papers  (2020-01-31T17:47:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.