MCN-CL: Multimodal Cross-Attention Network and Contrastive Learning for Multimodal Emotion Recognition

• URL: http://arxiv.org/abs/2511.10892v1
• Date: Fri, 14 Nov 2025 02:13:31 GMT
• Title: MCN-CL: Multimodal Cross-Attention Network and Contrastive Learning for Multimodal Emotion Recognition
• Authors: Feng Li, Ke Wu, Yongwei Li,
• Abstract summary: This paper proposes Multimodal Cross-Attention Network and Contrastive Learning (MCN-CL) for multimodal emotion recognition.<n>It uses a triple query mechanism and hard negative mining strategy to remove feature redundancy while preserving important emotional cues.<n>Experiment results on the IEMOCAP and MELD datasets show that our proposed method outperforms state-of-the-art approaches.
• Score: 8.732416479560605
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multimodal emotion recognition plays a key role in many domains, including mental health monitoring, educational interaction, and human-computer interaction. However, existing methods often face three major challenges: unbalanced category distribution, the complexity of dynamic facial action unit time modeling, and the difficulty of feature fusion due to modal heterogeneity. With the explosive growth of multimodal data in social media scenarios, the need for building an efficient cross-modal fusion framework for emotion recognition is becoming increasingly urgent. To this end, this paper proposes Multimodal Cross-Attention Network and Contrastive Learning (MCN-CL) for multimodal emotion recognition. It uses a triple query mechanism and hard negative mining strategy to remove feature redundancy while preserving important emotional cues, effectively addressing the issues of modal heterogeneity and category imbalance. Experiment results on the IEMOCAP and MELD datasets show that our proposed method outperforms state-of-the-art approaches, with Weighted F1 scores improving by 3.42% and 5.73%, respectively.

Related papers

• HyPCA-Net: Advancing Multimodal Fusion in Medical Image Analysis [0.0]
  We propose a Hybrid Parallel-Fusion Cascaded Attention Network (HyPCA-Net)<n>HyPCA-Net is composed of two core novel blocks: (a) a computationally efficient residual adaptive learning attention block for capturing modality-specific representations, and (b) a dual-view cascaded attention block aimed at learning robust shared representations across diverse modalities.<n>Experiments show that HyPCA-Net significantly outperforms existing leading methods, with improvements of up to 5.2% in performance and reductions of up to 73.1% in computational cost.
  arXiv  Detail & Related papers  (2026-02-18T07:47:49Z)
• From Sparse Decisions to Dense Reasoning: A Multi-attribute Trajectory Paradigm for Multimodal Moderation [59.27094165576015]
  We propose a novel learning paradigm (UniMod) that transitions from sparse decision-making to dense reasoning traces.<n>By constructing structured trajectories encompassing evidence grounding, modality assessment, risk mapping, policy decision, and response generation, we reformulate monolithic decision tasks into a multi-dimensional boundary learning process.<n>We introduce specialized optimization strategies to decouple task-specific parameters and rebalance training dynamics, effectively resolving interference between diverse objectives in multi-task learning.
  arXiv  Detail & Related papers  (2026-01-28T09:29:40Z)
• MCIHN: A Hybrid Network Model Based on Multi-path Cross-modal Interaction for Multimodal Emotion Recognition [7.944119407791842]
  We propose a hybrid network model based on multipath cross-modal interaction (MCIHN)<n> adversarial autoencoders (AAE) are constructed separately for each modality.<n> latent codes are fed into a predefined Cross-modal Gate Mechanism model (CGMM)
  arXiv  Detail & Related papers  (2025-10-28T16:04:03Z)
• GIA-MIC: Multimodal Emotion Recognition with Gated Interactive Attention and Modality-Invariant Learning Constraints [24.242098942377574]
  Multimodal emotion recognition (MER) extracts emotions from multimodal data, including visual, speech, and text inputs, playing a key role in human-computer interaction.<n>We propose a gated interactive attention mechanism to adaptively extract modality-specific features while enhancing emotional information through pairwise interactions.<n> Experiments on IEMOCAP demonstrate that our method outperforms state-of-the-art MER approaches, achieving WA 80.7% and UA 81.3%.
  arXiv  Detail & Related papers  (2025-06-01T07:07:02Z)
• A Novel Approach to for Multimodal Emotion Recognition : Multimodal semantic information fusion [3.1409950035735914]
  This paper proposes a novel multimodal emotion recognition approach, DeepMSI-MER, based on the integration of contrastive learning and visual sequence compression.<n> Experimental results on two public datasets, IEMOCAP and MELD, demonstrate that DeepMSI-MER significantly improves the accuracy and robustness of emotion recognition.
  arXiv  Detail & Related papers  (2025-02-12T17:07:43Z)
• WavFusion: Towards wav2vec 2.0 Multimodal Speech Emotion Recognition [2.3367170233149324]
  We propose WavFusion, a multimodal speech emotion recognition framework.<n>WavFusion addresses critical research problems in effective multimodal fusion, among modalities, and discriminative representation learning.<n>Our work highlights the importance of capturing nuanced cross-modal interactions and learning discriminative representations for accurate multimodal SER.
  arXiv  Detail & Related papers  (2024-12-07T06:43:39Z)
• MMA-DFER: MultiModal Adaptation of unimodal models for Dynamic Facial Expression Recognition in-the-wild [81.32127423981426]
  Multimodal emotion recognition based on audio and video data is important for real-world applications. Recent methods have focused on exploiting advances of self-supervised learning (SSL) for pre-training of strong multimodal encoders. We propose a different perspective on the problem and investigate the advancement of multimodal DFER performance by adapting SSL-pre-trained disjoint unimodal encoders.
  arXiv  Detail & Related papers  (2024-04-13T13:39:26Z)
• Joint Multimodal Transformer for Emotion Recognition in the Wild [49.735299182004404]
  Multimodal emotion recognition (MMER) systems typically outperform unimodal systems. This paper proposes an MMER method that relies on a joint multimodal transformer (JMT) for fusion with key-based cross-attention.
  arXiv  Detail & Related papers  (2024-03-15T17:23:38Z)
• Deep Imbalanced Learning for Multimodal Emotion Recognition in Conversations [15.705757672984662]
  Multimodal Emotion Recognition in Conversations (MERC) is a significant development direction for machine intelligence. Many data in MERC naturally exhibit an imbalanced distribution of emotion categories, and researchers ignore the negative impact of imbalanced data on emotion recognition. We propose the Class Boundary Enhanced Representation Learning (CBERL) model to address the imbalanced distribution of emotion categories in raw data. We have conducted extensive experiments on the IEMOCAP and MELD benchmark datasets, and the results show that CBERL has achieved a certain performance improvement in the effectiveness of emotion recognition.
  arXiv  Detail & Related papers  (2023-12-11T12:35:17Z)
• Cross-Attention is Not Enough: Incongruity-Aware Dynamic Hierarchical Fusion for Multimodal Affect Recognition [69.32305810128994]
  Incongruity between modalities poses a challenge for multimodal fusion, especially in affect recognition. We propose the Hierarchical Crossmodal Transformer with Dynamic Modality Gating (HCT-DMG), a lightweight incongruity-aware model. HCT-DMG: 1) outperforms previous multimodal models with a reduced size of approximately 0.8M parameters; 2) recognizes hard samples where incongruity makes affect recognition difficult; 3) mitigates the incongruity at the latent level in crossmodal attention.
  arXiv  Detail & Related papers  (2023-05-23T01:24:15Z)
• Group Gated Fusion on Attention-based Bidirectional Alignment for Multimodal Emotion Recognition [63.07844685982738]
  This paper presents a new model named as Gated Bidirectional Alignment Network (GBAN), which consists of an attention-based bidirectional alignment network over LSTM hidden states. We empirically show that the attention-aligned representations outperform the last-hidden-states of LSTM significantly. The proposed GBAN model outperforms existing state-of-the-art multimodal approaches on the IEMOCAP dataset.
  arXiv  Detail & Related papers  (2022-01-17T09:46:59Z)
• Channel Exchanging Networks for Multimodal and Multitask Dense Image Prediction [125.18248926508045]
  We propose Channel-Exchanging-Network (CEN) which is self-adaptive, parameter-free, and more importantly, applicable for both multimodal fusion and multitask learning. CEN dynamically exchanges channels betweenworks of different modalities. For the application of dense image prediction, the validity of CEN is tested by four different scenarios.
  arXiv  Detail & Related papers  (2021-12-04T05:47:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.