Dual-branch Graph Domain Adaptation for Cross-scenario Multi-modal Emotion Recognition

• URL: http://arxiv.org/abs/2603.26840v1
• Date: Fri, 27 Mar 2026 08:21:09 GMT
• Title: Dual-branch Graph Domain Adaptation for Cross-scenario Multi-modal Emotion Recognition
• Abstract summary: Multimodal Emotion Recognition in Conversations (MERC) aims to predict speakers' emotional states in multi-turn dialogues through text, audio, and visual cues.<n>Existing MERC methods generally neglect cross-scenario variations, limiting their ability to transfer models trained on a source domain to unseen target domains.<n>We propose a Dual-branch Graph Adaptation framework (DGDA) for multimodal emotion recognition under cross-scenario conditions.
• Score: 43.17664347937779
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multimodal Emotion Recognition in Conversations (MERC) aims to predict speakers' emotional states in multi-turn dialogues through text, audio, and visual cues. In real-world settings, conversation scenarios differ significantly in speakers, topics, styles, and noise levels. Existing MERC methods generally neglect these cross-scenario variations, limiting their ability to transfer models trained on a source domain to unseen target domains. To address this issue, we propose a Dual-branch Graph Domain Adaptation framework (DGDA) for multimodal emotion recognition under cross-scenario conditions. We first construct an emotion interaction graph to characterize complex emotional dependencies among utterances. A dual-branch encoder, consisting of a hypergraph neural network (HGNN) and a path neural network (PathNN), is then designed to explicitly model multivariate relationships and implicitly capture global dependencies. To enable out-of-domain generalization, a domain adversarial discriminator is introduced to learn invariant representations across domains. Furthermore, a regularization loss is incorporated to suppress the negative influence of noisy labels. To the best of our knowledge, DGDA is the first MERC framework that jointly addresses domain shift and label noise. Theoretical analysis provides tighter generalization bounds, and extensive experiments on IEMOCAP and MELD demonstrate that DGDA consistently outperforms strong baselines and better adapts to cross-scenario conversations. Our code is available at https://github.com/Xudmm1239439/DGDA-Net.

Related papers

• Disentangled Dual-Branch Graph Learning for Conversational Emotion Recognition [37.30058027224552]
  Multimodal emotion recognition in conversations aims to infer utterance-level emotions by jointly modeling textual, acoustic, and visual cues within context.<n>We propose a framework that combines dual-space feature disentanglement with dual-branch graph learning.<n> Experiments on IEMOCAP and MELD demonstrate that the proposed method achieves superior performance over strong baselines.
  arXiv  Detail & Related papers  (2026-04-03T14:47:26Z)
• Knowledge-Aligned Counterfactual-Enhancement Diffusion Perception for Unsupervised Cross-Domain Visual Emotion Recognition [23.396309161898465]
  Unsupervised Cross-Domain Visual Emotion Recognition (UCDVER) task aims to generalize visual emotion recognition from source domain to low-resource target domain.<n>To mitigate these issues, we propose the Knowledge-aligned Counterfactual-enhancement Diffusion Perception (KCDP) framework.<n>Our model surpasses SOTA models in both perceptibility and generalization, e.g., gaining 12% improvements over the SOTA VER model TGCA-PVT.
  arXiv  Detail & Related papers  (2025-05-26T08:50:30Z)
• Effective Context Modeling Framework for Emotion Recognition in Conversations [2.7175580940471913]
  Emotion Recognition in Conversations (ERC) facilitates a deeper understanding of the emotions conveyed by speakers in each utterance within a conversation.<n>Recent Graph Neural Networks (GNNs) have demonstrated their strengths in capturing data relationships.<n>We propose ConxGNN, a novel GNN-based framework designed to capture contextual information in conversations.
  arXiv  Detail & Related papers  (2024-12-21T02:22:06Z)
• Dynamic Graph Neural ODE Network for Multi-modal Emotion Recognition in Conversation [14.158939954453933]
  We propose a Dynamic Graph Neural Ordinary Differential Equation Network (DGODE) for Multimodal emotion recognition in conversation (MERC)<n>The proposed DGODE combines the dynamic changes of emotions to capture the temporal dependency of speakers' emotions.<n>Experiments on two publicly available multimodal emotion recognition datasets demonstrate that the proposed DGODE model has superior performance compared to various baselines.
  arXiv  Detail & Related papers  (2024-12-04T01:07:59Z)
• PanoSent: A Panoptic Sextuple Extraction Benchmark for Multimodal Conversational Aspect-based Sentiment Analysis [74.41260927676747]
  This paper bridges the gaps by introducing a multimodal conversational Sentiment Analysis (ABSA) To benchmark the tasks, we construct PanoSent, a dataset annotated both manually and automatically, featuring high quality, large scale, multimodality, multilingualism, multi-scenarios, and covering both implicit and explicit sentiment elements. To effectively address the tasks, we devise a novel Chain-of-Sentiment reasoning framework, together with a novel multimodal large language model (namely Sentica) and a paraphrase-based verification mechanism.
  arXiv  Detail & Related papers  (2024-08-18T13:51:01Z)
• Object Segmentation by Mining Cross-Modal Semantics [68.88086621181628]
  We propose a novel approach by mining the Cross-Modal Semantics to guide the fusion and decoding of multimodal features. Specifically, we propose a novel network, termed XMSNet, consisting of (1) all-round attentive fusion (AF), (2) coarse-to-fine decoder (CFD), and (3) cross-layer self-supervision.
  arXiv  Detail & Related papers  (2023-05-17T14:30:11Z)
• PoE: a Panel of Experts for Generalized Automatic Dialogue Assessment [58.46761798403072]
  A model-based automatic dialogue evaluation metric (ADEM) is expected to perform well across multiple domains. Despite significant progress, an ADEM that works well in one domain does not necessarily generalize to another. We propose a Panel of Experts (PoE) network that consists of a shared transformer encoder and a collection of lightweight adapters.
  arXiv  Detail & Related papers  (2022-12-18T02:26:50Z)
• Mix and Reason: Reasoning over Semantic Topology with Data Mixing for Domain Generalization [48.90173060487124]
  Domain generalization (DG) enables a learning machine from multiple seen source domains to an unseen target one. mire consists of two key components, namely, Category-aware Data Mixing (CDM) and Adaptive Semantic Topology Refinement (ASTR) experiments on multiple DG benchmarks validate the effectiveness and robustness of the proposed mire.
  arXiv  Detail & Related papers  (2022-10-14T06:52:34Z)
• Emotional Semantics-Preserved and Feature-Aligned CycleGAN for Visual Emotion Adaptation [85.20533077846606]
  Unsupervised domain adaptation (UDA) studies the problem of transferring models trained on one labeled source domain to another unlabeled target domain. In this paper, we focus on UDA in visual emotion analysis for both emotion distribution learning and dominant emotion classification. We propose a novel end-to-end cycle-consistent adversarial model, termed CycleEmotionGAN++.
  arXiv  Detail & Related papers  (2020-11-25T01:31:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.