Multi-modal Anchor Gated Transformer with Knowledge Distillation for Emotion Recognition in Conversation

• URL: http://arxiv.org/abs/2506.18716v1
• Date: Mon, 23 Jun 2025 14:53:22 GMT
• Title: Multi-modal Anchor Gated Transformer with Knowledge Distillation for Emotion Recognition in Conversation
• Authors: Jie Li, Shifei Ding, Lili Guo, Xuan Li,
• Abstract summary: Emotion Recognition in Conversation (ERC) aims to detect the emotions of individual utterances within a conversation.<n> generating efficient and modality-specific representations for each utterance remains a significant challenge.<n>We propose the Multi-modal Anchor Gated Transformer with Knowledge Distillation (MAGTKD) for the ERC task.
• Score: 16.83256715431888
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Emotion Recognition in Conversation (ERC) aims to detect the emotions of individual utterances within a conversation. Generating efficient and modality-specific representations for each utterance remains a significant challenge. Previous studies have proposed various models to integrate features extracted using different modality-specific encoders. However, they neglect the varying contributions of modalities to this task and introduce high complexity by aligning modalities at the frame level. To address these challenges, we propose the Multi-modal Anchor Gated Transformer with Knowledge Distillation (MAGTKD) for the ERC task. Specifically, prompt learning is employed to enhance textual modality representations, while knowledge distillation is utilized to strengthen representations of weaker modalities. Furthermore, we introduce a multi-modal anchor gated transformer to effectively integrate utterance-level representations across modalities. Extensive experiments on the IEMOCAP and MELD datasets demonstrate the effectiveness of knowledge distillation in enhancing modality representations and achieve state-of-the-art performance in emotion recognition. Our code is available at: https://github.com/JieLi-dd/MAGTKD.

Related papers

• VAEmo: Efficient Representation Learning for Visual-Audio Emotion with Knowledge Injection [50.57849622045192]
  We propose VAEmo, an efficient framework for emotion-centric joint VA representation learning with external knowledge injection.<n>VAEmo achieves state-of-the-art performance with a compact design, highlighting the benefit of unified cross-modal encoding and emotion-aware semantic guidance.
  arXiv  Detail & Related papers  (2025-05-05T03:00:51Z)
• Mamba-Enhanced Text-Audio-Video Alignment Network for Emotion Recognition in Conversations [15.748798247815298]
  We propose a novel Mamba-enhanced Text-Audio-Video alignment network (MaTAV) for the Emotion Recognition in Conversations (ERC) task. MaTAV is with the advantages of aligning unimodal features to ensure consistency across different modalities and handling long input sequences to better capture contextual multimodal information.
  arXiv  Detail & Related papers  (2024-09-08T23:09:22Z)
• AIMDiT: Modality Augmentation and Interaction via Multimodal Dimension Transformation for Emotion Recognition in Conversations [57.99479708224221]
  We propose a novel framework called AIMDiT to solve the problem of multimodal fusion of deep features. Experiments conducted using our AIMDiT framework on the public benchmark dataset MELD reveal 2.34% and 2.87% improvements in terms of the Acc-7 and w-F1 metrics.
  arXiv  Detail & Related papers  (2024-04-12T11:31:18Z)
• A Transformer-Based Model With Self-Distillation for Multimodal Emotion Recognition in Conversations [15.77747948751497]
  We propose a transformer-based model with self-distillation (SDT) for the task. The proposed model captures intra- and inter-modal interactions by utilizing intra- and inter-modal transformers. We introduce self-distillation to transfer knowledge of hard and soft labels from the proposed model to each modality.
  arXiv  Detail & Related papers  (2023-10-31T14:33:30Z)
• Exploiting Modality-Specific Features For Multi-Modal Manipulation Detection And Grounding [54.49214267905562]
  We construct a transformer-based framework for multi-modal manipulation detection and grounding tasks. Our framework simultaneously explores modality-specific features while preserving the capability for multi-modal alignment. We propose an implicit manipulation query (IMQ) that adaptively aggregates global contextual cues within each modality.
  arXiv  Detail & Related papers  (2023-09-22T06:55:41Z)
• Multimodal Emotion Recognition using Transfer Learning from Speaker Recognition and BERT-based models [53.31917090073727]
  We propose a neural network-based emotion recognition framework that uses a late fusion of transfer-learned and fine-tuned models from speech and text modalities. We evaluate the effectiveness of our proposed multimodal approach on the interactive emotional dyadic motion capture dataset.
  arXiv  Detail & Related papers  (2022-02-16T00:23:42Z)
• MEmoBERT: Pre-training Model with Prompt-based Learning for Multimodal Emotion Recognition [118.73025093045652]
  We propose a pre-training model textbfMEmoBERT for multimodal emotion recognition. Unlike the conventional "pre-train, finetune" paradigm, we propose a prompt-based method that reformulates the downstream emotion classification task as a masked text prediction. Our proposed MEmoBERT significantly enhances emotion recognition performance.
  arXiv  Detail & Related papers  (2021-10-27T09:57:00Z)
• Fusion with Hierarchical Graphs for Mulitmodal Emotion Recognition [7.147235324895931]
  This paper proposes a novel hierarchical graph network (HFGCN) model that learns more informative multimodal representations. Specifically, the proposed model fuses multimodality inputs using a two-stage graph construction approach and encodes the modality dependencies into the conversation representation. Experiments showed the effectiveness of our proposed model for more accurate AER, which yielded state-of-the-art results on two public datasets.
  arXiv  Detail & Related papers  (2021-09-15T08:21:01Z)
• Distilling Audio-Visual Knowledge by Compositional Contrastive Learning [51.20935362463473]
  We learn a compositional embedding that closes the cross-modal semantic gap. We establish a new, comprehensive multi-modal distillation benchmark on three video datasets.
  arXiv  Detail & Related papers  (2021-04-22T09:31:20Z)
• Self-Supervised learning with cross-modal transformers for emotion recognition [20.973999078271483]
  Self-supervised learning has shown improvements on tasks with limited labeled datasets in domains like speech and natural language. In this work, we extend self-supervised training to multi-modal applications.
  arXiv  Detail & Related papers  (2020-11-20T21:38:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.