CMCRD: Cross-Modal Contrastive Representation Distillation for Emotion Recognition

• URL: http://arxiv.org/abs/2504.09221v1
• Date: Sat, 12 Apr 2025 13:56:20 GMT
• Title: CMCRD: Cross-Modal Contrastive Representation Distillation for Emotion Recognition
• Authors: Siyuan Kan, Huanyu Wu, Zhenyao Cui, Fan Huang, Xiaolong Xu, Dongrui Wu,
• Abstract summary: Cross-modal contrastive representation distillation (CMCRD) uses a pre-trained eye movement classification model to assist the training of an EEG classification model.<n>During test, only EEG signals (or eye movement signals) are acquired, eliminating the need for multi-modal data.<n>Compared with the EEG-only model, it improved the average classification accuracy by about 6.2%.
• Score: 15.72347392139404
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Emotion recognition is an important component of affective computing, and also human-machine interaction. Unimodal emotion recognition is convenient, but the accuracy may not be high enough; on the contrary, multi-modal emotion recognition may be more accurate, but it also increases the complexity and cost of the data collection system. This paper considers cross-modal emotion recognition, i.e., using both electroencephalography (EEG) and eye movement in training, but only EEG or eye movement in test. We propose cross-modal contrastive representation distillation (CMCRD), which uses a pre-trained eye movement classification model to assist the training of an EEG classification model, improving feature extraction from EEG signals, or vice versa. During test, only EEG signals (or eye movement signals) are acquired, eliminating the need for multi-modal data. CMCRD not only improves the emotion recognition accuracy, but also makes the system more simplified and practical. Experiments using three different neural network architectures on three multi-modal emotion recognition datasets demonstrated the effectiveness of CMCRD. Compared with the EEG-only model, it improved the average classification accuracy by about 6.2%.

Related papers

• learning discriminative features from spectrograms using center loss for speech emotion recognition [62.13177498013144]
  We propose a novel approach to learn discriminative features from variable length spectrograms for emotion recognition. The softmax cross-entropy loss enables features from different emotion categories separable, and the center loss efficiently pulls the features belonging to the same emotion category to their center.
  arXiv  Detail & Related papers  (2025-01-02T06:52:28Z)
• Smile upon the Face but Sadness in the Eyes: Emotion Recognition based on Facial Expressions and Eye Behaviors [63.194053817609024]
  We introduce eye behaviors as an important emotional cues for the creation of a new Eye-behavior-aided Multimodal Emotion Recognition dataset. For the first time, we provide annotations for both Emotion Recognition (ER) and Facial Expression Recognition (FER) in the EMER dataset. We specifically design a new EMERT architecture to concurrently enhance performance in both ER and FER.
  arXiv  Detail & Related papers  (2024-11-08T04:53:55Z)
• A Supervised Information Enhanced Multi-Granularity Contrastive Learning Framework for EEG Based Emotion Recognition [14.199298112101802]
  This study introduces a novel Supervised Info-enhanced Contrastive Learning framework for EEG based Emotion Recognition (SICLEER) We propose a joint learning model combining self-supervised contrastive learning loss and supervised classification loss.
  arXiv  Detail & Related papers  (2024-05-12T11:51:00Z)
• Joint Contrastive Learning with Feature Alignment for Cross-Corpus EEG-based Emotion Recognition [2.1645626994550664]
  We propose a novel Joint Contrastive learning framework with Feature Alignment to address cross-corpus EEG-based emotion recognition. In the pre-training stage, a joint domain contrastive learning strategy is introduced to characterize generalizable time-frequency representations of EEG signals. In the fine-tuning stage, JCFA is refined in conjunction with downstream tasks, where the structural connections among brain electrodes are considered.
  arXiv  Detail & Related papers  (2024-04-15T08:21:17Z)
• Emotion Detection from EEG using Transfer Learning [0.0]
  We employed transfer learning to overcome the challenge of limited data availability in EEG-based emotion detection. The input to the model was in the form of an image matrix, which comprised Mean Phase Coherence (MPC) and Magnitude Squared Coherence (MSC)
  arXiv  Detail & Related papers  (2023-06-09T05:43:06Z)
• EEG2Vec: Learning Affective EEG Representations via Variational Autoencoders [27.3162026528455]
  We explore whether representing neural data, in response to emotional stimuli, in a latent vector space can serve to both predict emotional states. We propose a conditional variational autoencoder based framework, EEG2Vec, to learn generative-discriminative representations from EEG data.
  arXiv  Detail & Related papers  (2022-07-16T19:25:29Z)
• Multimodal Emotion Recognition using Transfer Learning from Speaker Recognition and BERT-based models [53.31917090073727]
  We propose a neural network-based emotion recognition framework that uses a late fusion of transfer-learned and fine-tuned models from speech and text modalities. We evaluate the effectiveness of our proposed multimodal approach on the interactive emotional dyadic motion capture dataset.
  arXiv  Detail & Related papers  (2022-02-16T00:23:42Z)
• Emotional EEG Classification using Connectivity Features and Convolutional Neural Networks [81.74442855155843]
  We introduce a new classification system that utilizes brain connectivity with a CNN and validate its effectiveness via the emotional video classification. The level of concentration of the brain connectivity related to the emotional property of the target video is correlated with classification performance.
  arXiv  Detail & Related papers  (2021-01-18T13:28:08Z)
• A Novel Transferability Attention Neural Network Model for EEG Emotion Recognition [51.203579838210885]
  We propose a transferable attention neural network (TANN) for EEG emotion recognition. TANN learns the emotional discriminative information by highlighting the transferable EEG brain regions data and samples adaptively. This can be implemented by measuring the outputs of multiple brain-region-level discriminators and one single sample-level discriminator.
  arXiv  Detail & Related papers  (2020-09-21T02:42:30Z)
• Investigating EEG-Based Functional Connectivity Patterns for Multimodal Emotion Recognition [8.356765961526955]
  We investigate three functional connectivity network features: strength, clustering, coefficient and eigenvector centrality. The discrimination ability of the EEG connectivity features in emotion recognition is evaluated on three public EEG datasets. We construct a multimodal emotion recognition model by combining the functional connectivity features from EEG and the features from eye movements or physiological signals.
  arXiv  Detail & Related papers  (2020-04-04T16:51:56Z)
• Continuous Emotion Recognition via Deep Convolutional Autoencoder and Support Vector Regressor [70.2226417364135]
  It is crucial that the machine should be able to recognize the emotional state of the user with high accuracy. Deep neural networks have been used with great success in recognizing emotions. We present a new model for continuous emotion recognition based on facial expression recognition.
  arXiv  Detail & Related papers  (2020-01-31T17:47:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.