Cross-modal Context Fusion and Adaptive Graph Convolutional Network for Multimodal Conversational Emotion Recognition

• URL: http://arxiv.org/abs/2501.15063v1
• Date: Sat, 25 Jan 2025 03:53:53 GMT
• Title: Cross-modal Context Fusion and Adaptive Graph Convolutional Network for Multimodal Conversational Emotion Recognition
• Authors: Junwei Feng, Xueyan Fan,
• Abstract summary: This article proposes a new multimodal emotion recognition method, including a cross modal context fusion module, an adaptive graph convolutional encoding module, and an emotion classification module.<n>Our model has surpassed some state-of-the-art methods on publicly available benchmark datasets and achieved high recognition accuracy.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Emotion recognition has a wide range of applications in human-computer interaction, marketing, healthcare, and other fields. In recent years, the development of deep learning technology has provided new methods for emotion recognition. Prior to this, many emotion recognition methods have been proposed, including multimodal emotion recognition methods, but these methods ignore the mutual interference between different input modalities and pay little attention to the directional dialogue between speakers. Therefore, this article proposes a new multimodal emotion recognition method, including a cross modal context fusion module, an adaptive graph convolutional encoding module, and an emotion classification module. The cross modal context module includes a cross modal alignment module and a context fusion module, which are used to reduce the noise introduced by mutual interference between different input modalities. The adaptive graph convolution module constructs a dialogue relationship graph for extracting dependencies and self dependencies between speakers. Our model has surpassed some state-of-the-art methods on publicly available benchmark datasets and achieved high recognition accuracy.

Related papers

• Heterogeneous bimodal attention fusion for speech emotion recognition [14.24203981584309]
  We propose a novel framework called Heterogeneous Bimodal Attention Fusion (HBAF) for multi-level multi-modal interaction in conversational emotion recognition. The uni-modal representation module incorporates contextual content into low-level audio representations to bridge the heterogeneous multi-modal gap. The multi-modal fusion module uses dynamic bimodal attention and a dynamic gating mechanism to filter incorrect cross-modal relationships.
  arXiv  Detail & Related papers  (2025-03-09T02:50:49Z)
• Hierarchical Banzhaf Interaction for General Video-Language Representation Learning [60.44337740854767]
  Multimodal representation learning plays an important role in the artificial intelligence domain.<n>We introduce a new approach that models video-text as game players using multivariate cooperative game theory.<n>We extend our original structure into a flexible encoder-decoder framework, enabling the model to adapt to various downstream tasks.
  arXiv  Detail & Related papers  (2024-12-30T14:09:15Z)
• Effective Context Modeling Framework for Emotion Recognition in Conversations [2.7175580940471913]
  Emotion Recognition in Conversations (ERC) facilitates a deeper understanding of the emotions conveyed by speakers in each utterance within a conversation.<n>Recent Graph Neural Networks (GNNs) have demonstrated their strengths in capturing data relationships.<n>We propose ConxGNN, a novel GNN-based framework designed to capture contextual information in conversations.
  arXiv  Detail & Related papers  (2024-12-21T02:22:06Z)
• AIMDiT: Modality Augmentation and Interaction via Multimodal Dimension Transformation for Emotion Recognition in Conversations [57.99479708224221]
  We propose a novel framework called AIMDiT to solve the problem of multimodal fusion of deep features. Experiments conducted using our AIMDiT framework on the public benchmark dataset MELD reveal 2.34% and 2.87% improvements in terms of the Acc-7 and w-F1 metrics.
  arXiv  Detail & Related papers  (2024-04-12T11:31:18Z)
• Joint Multimodal Transformer for Emotion Recognition in the Wild [49.735299182004404]
  Multimodal emotion recognition (MMER) systems typically outperform unimodal systems. This paper proposes an MMER method that relies on a joint multimodal transformer (JMT) for fusion with key-based cross-attention.
  arXiv  Detail & Related papers  (2024-03-15T17:23:38Z)
• AMuSE: Adaptive Multimodal Analysis for Speaker Emotion Recognition in Group Conversations [39.79734528362605]
  Multimodal Attention Network captures cross-modal interactions at various levels of spatial abstraction. AMuSE model condenses both spatial and temporal features into two dense descriptors: speaker-level and utterance-level.
  arXiv  Detail & Related papers  (2024-01-26T19:17:05Z)
• From Text to Pixels: A Context-Aware Semantic Synergy Solution for Infrared and Visible Image Fusion [66.33467192279514]
  We introduce a text-guided multi-modality image fusion method that leverages the high-level semantics from textual descriptions to integrate semantics from infrared and visible images. Our method not only produces visually superior fusion results but also achieves a higher detection mAP over existing methods, achieving state-of-the-art results.
  arXiv  Detail & Related papers  (2023-12-31T08:13:47Z)
• Adversarial Representation with Intra-Modal and Inter-Modal Graph Contrastive Learning for Multimodal Emotion Recognition [14.639340916340801]
  We propose a novel Adversarial Representation with Intra-Modal and Inter-Modal Graph Contrastive for Multimodal Emotion Recognition (AR-IIGCN) method. Firstly, we input video, audio, and text features into a multi-layer perceptron (MLP) to map them into separate feature spaces. Secondly, we build a generator and a discriminator for the three modal features through adversarial representation. Thirdly, we introduce contrastive graph representation learning to capture intra-modal and inter-modal complementary semantic information.
  arXiv  Detail & Related papers  (2023-12-28T01:57:26Z)
• Modeling Motion with Multi-Modal Features for Text-Based Video Segmentation [56.41614987789537]
  Text-based video segmentation aims to segment the target object in a video based on a describing sentence. We propose a method to fuse and align appearance, motion, and linguistic features to achieve accurate segmentation.
  arXiv  Detail & Related papers  (2022-04-06T02:42:33Z)
• A Joint Cross-Attention Model for Audio-Visual Fusion in Dimensional Emotion Recognition [46.443866373546726]
  We focus on dimensional emotion recognition based on the fusion of facial and vocal modalities extracted from videos. We propose a joint cross-attention model that relies on the complementary relationships to extract the salient features. Our proposed A-V fusion model provides a cost-effective solution that can outperform state-of-the-art approaches.
  arXiv  Detail & Related papers  (2022-03-28T14:09:43Z)
• High-Modality Multimodal Transformer: Quantifying Modality & Interaction Heterogeneity for High-Modality Representation Learning [112.51498431119616]
  This paper studies efficient representation learning for high-modality scenarios involving a large set of diverse modalities. A single model, HighMMT, scales up to 10 modalities (text, image, audio, video, sensors, proprioception, speech, time-series, sets, and tables) and 15 tasks from 5 research areas.
  arXiv  Detail & Related papers  (2022-03-02T18:56:20Z)
• Multimodal Emotion Recognition using Transfer Learning from Speaker Recognition and BERT-based models [53.31917090073727]
  We propose a neural network-based emotion recognition framework that uses a late fusion of transfer-learned and fine-tuned models from speech and text modalities. We evaluate the effectiveness of our proposed multimodal approach on the interactive emotional dyadic motion capture dataset.
  arXiv  Detail & Related papers  (2022-02-16T00:23:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.