PHemoNet: A Multimodal Network for Physiological Signals

• URL: http://arxiv.org/abs/2410.00010v1
• Date: Fri, 13 Sep 2024 21:14:27 GMT
• Title: PHemoNet: A Multimodal Network for Physiological Signals
• Authors: Eleonora Lopez, Aurelio Uncini, Danilo Comminiello,
• Abstract summary: We introduce PHemoNet, a fully hypercomplex network for multimodal emotion recognition from physiological signals. The architecture comprises modality-specific encoders and a fusion module. The proposed method outperforms current state-of-the-art models on the MAHNOB-HCI dataset.
• Score: 9.54382727022316
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Emotion recognition is essential across numerous fields, including medical applications and brain-computer interface (BCI). Emotional responses include behavioral reactions, such as tone of voice and body movement, and changes in physiological signals, such as the electroencephalogram (EEG). The latter are involuntary, thus they provide a reliable input for identifying emotions, in contrast to the former which individuals can consciously control. These signals reveal true emotional states without intentional alteration, thus increasing the accuracy of emotion recognition models. However, multimodal deep learning methods from physiological signals have not been significantly investigated. In this paper, we introduce PHemoNet, a fully hypercomplex network for multimodal emotion recognition from physiological signals. In detail, the architecture comprises modality-specific encoders and a fusion module. Both encoders and fusion modules are defined in the hypercomplex domain through parameterized hypercomplex multiplications (PHMs) that can capture latent relations between the different dimensions of each modality and between the modalities themselves. The proposed method outperforms current state-of-the-art models on the MAHNOB-HCI dataset in classifying valence and arousal using electroencephalograms (EEGs) and peripheral physiological signals. The code for this work is available at https://github.com/ispamm/MHyEEG.

Related papers

• Online Multi-level Contrastive Representation Distillation for Cross-Subject fNIRS Emotion Recognition [11.72499878247794]
  We propose a novel cross-subject fNIRS emotion recognition method, called the Online Multi-level Contrastive Representation Distillation framework (OMCRD) OMCRD is a framework designed for mutual learning among multiple lightweight student networks. Some experimental results demonstrate that OMCRD achieves state-of-the-art results in emotional perception and affective imagery tasks.
  arXiv  Detail & Related papers  (2024-09-24T13:30:15Z)
• Hierarchical Hypercomplex Network for Multimodal Emotion Recognition [9.54382727022316]
  We introduce a fully hypercomplex network with a hierarchical learning structure to fully capture correlations. The proposed architecture surpasses state-of-the-art models on the MAHNOB-HCI dataset for emotion recognition.
  arXiv  Detail & Related papers  (2024-09-13T21:07:49Z)
• Decoding Human Emotions: Analyzing Multi-Channel EEG Data using LSTM Networks [0.0]
  This study aims to understand and improve the predictive accuracy of emotional state classification by applying a Long Short-Term Memory (LSTM) network to analyze EEG signals. Using a popular dataset of multi-channel EEG recordings known as DEAP, we look towards leveraging LSTM networks' properties to handle temporal dependencies within EEG signal data. We obtain accuracies of 89.89%, 90.33%, 90.70%, and 90.54% for arousal, valence, dominance, and likeness, respectively, demonstrating significant improvements in emotion recognition model capabilities.
  arXiv  Detail & Related papers  (2024-08-19T18:10:47Z)
• Multi-modal Mood Reader: Pre-trained Model Empowers Cross-Subject Emotion Recognition [23.505616142198487]
  We develop a Pre-trained model based Multimodal Mood Reader for cross-subject emotion recognition. The model learns universal latent representations of EEG signals through pre-training on large scale dataset. Extensive experiments on public datasets demonstrate Mood Reader's superior performance in cross-subject emotion recognition tasks.
  arXiv  Detail & Related papers  (2024-05-28T14:31:11Z)
• MindFormer: Semantic Alignment of Multi-Subject fMRI for Brain Decoding [50.55024115943266]
  We introduce a novel semantic alignment method of multi-subject fMRI signals using so-called MindFormer. This model is specifically designed to generate fMRI-conditioned feature vectors that can be used for conditioning Stable Diffusion model for fMRI- to-image generation or large language model (LLM) for fMRI-to-text generation. Our experimental results demonstrate that MindFormer generates semantically consistent images and text across different subjects.
  arXiv  Detail & Related papers  (2024-05-28T00:36:25Z)
• Hypercomplex Multimodal Emotion Recognition from EEG and Peripheral Physiological Signals [7.293063257956068]
  We propose a hypercomplex multimodal network equipped with a novel fusion module comprising parameterized hypercomplex multiplications. We perform classification of valence and arousal from electroencephalogram (EEG) and peripheral physiological signals, employing the publicly available database MAHNOB-HCI.
  arXiv  Detail & Related papers  (2023-10-11T16:45:44Z)
• fMRI from EEG is only Deep Learning away: the use of interpretable DL to unravel EEG-fMRI relationships [68.8204255655161]
  We present an interpretable domain grounded solution to recover the activity of several subcortical regions from multichannel EEG data. We recover individual spatial and time-frequency patterns of scalp EEG predictive of the hemodynamic signal in the subcortical nuclei.
  arXiv  Detail & Related papers  (2022-10-23T15:11:37Z)
• Multimodal Emotion Recognition using Transfer Learning from Speaker Recognition and BERT-based models [53.31917090073727]
  We propose a neural network-based emotion recognition framework that uses a late fusion of transfer-learned and fine-tuned models from speech and text modalities. We evaluate the effectiveness of our proposed multimodal approach on the interactive emotional dyadic motion capture dataset.
  arXiv  Detail & Related papers  (2022-02-16T00:23:42Z)
• Emotional EEG Classification using Connectivity Features and Convolutional Neural Networks [81.74442855155843]
  We introduce a new classification system that utilizes brain connectivity with a CNN and validate its effectiveness via the emotional video classification. The level of concentration of the brain connectivity related to the emotional property of the target video is correlated with classification performance.
  arXiv  Detail & Related papers  (2021-01-18T13:28:08Z)
• A Novel Transferability Attention Neural Network Model for EEG Emotion Recognition [51.203579838210885]
  We propose a transferable attention neural network (TANN) for EEG emotion recognition. TANN learns the emotional discriminative information by highlighting the transferable EEG brain regions data and samples adaptively. This can be implemented by measuring the outputs of multiple brain-region-level discriminators and one single sample-level discriminator.
  arXiv  Detail & Related papers  (2020-09-21T02:42:30Z)
• Video-based Remote Physiological Measurement via Cross-verified Feature Disentangling [121.50704279659253]
  We propose a cross-verified feature disentangling strategy to disentangle the physiological features with non-physiological representations. We then use the distilled physiological features for robust multi-task physiological measurements. The disentangled features are finally used for the joint prediction of multiple physiological signals like average HR values and r signals.
  arXiv  Detail & Related papers  (2020-07-16T09:39:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.