EmoEUS: Uncertainty Supervision for Multimodal Emotion Recognition in Conversation

• URL: http://arxiv.org/abs/2607.18336v1
• Date: Sun, 19 Jul 2026 18:09:22 GMT
• Title: EmoEUS: Uncertainty Supervision for Multimodal Emotion Recognition in Conversation
• Abstract summary: We propose EmoEUS, an explicit uncertainty supervision framework for multimodal emotion recognition.<n>EmoEUS performs uncertainty-aware multimodal fusion by dynamically weighting modalities using learned variance estimates.<n>Experiments on IEMOCAP and MELD show that EmoEUS consistently outperforms state-of-the-art methods.
• Score: 58.055014746373054
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multimodal emotion recognition in conversation (MERC) can leverage multimodal and contextual cues to boost recognition performance. However, existing fusion approaches in MERC often ignore modality-specific uncertainty across utterances caused by conflicting cues, varying noise, and missing modality-specific signals. We propose EmoEUS, an explicit uncertainty supervision framework for MERC. EmoEUS performs uncertainty-aware multimodal fusion by dynamically weighting modalities using learned variance estimates. We also introduce an explicitly supervised loss that aligns each utterance's predicted variance with the distance between the utterance's distributional representation and its emotion- and modality-specific cluster center. Experiments on IEMOCAP and MELD show that EmoEUS consistently outperforms state-of-the-art methods.

Related papers

• EII-SCL: Harnessing Emotional Inertia for Multimodal Emotion Recognition in Conversation [51.04910642896281]
  Multimodal emotion recognition in conversation (MERC) achieves accurate predictions by integrating multimodal and contextual information in dialogues.<n>Current MERC approaches overlook the impact of contextual emotional inertia in emotion shift, leading to sub-optimal performance.<n>We propose a novel Emotional Inertia-Informed Supervised Contrastive Learning module (EII-SCL) that informs the contrastive objective by constructing inertia-affected samples within temporal windows.
  arXiv  Detail & Related papers  (2026-07-19T18:09:54Z)
• MRUF: Multi-granularity Routing with Uncertainty-Aware Fusion for Robust Multimodal Sentiment Analysis [20.73891715749963]
  We propose a reliability-aware fusion method that combines multi-granularity routing with uncertainty-aware calibration.<n> MRUF summarizes sentiment-relevant representations, performs subspace- and modality-level routing, and supervises modality routing with leave-one-out error increases to estimate utterance-level modality importance.
  arXiv  Detail & Related papers  (2026-07-12T06:32:54Z)
• EmoMM: Benchmarking and Steering MLLM for Multimodal Emotion Recognition under Conflict and Missingness [17.292747703126796]
  We introduce EmoMM, a comprehensive benchmark featuring modality-aligned, conflict, and missing subsets.<n>We propose Conflict-aware Head-level Attention Steering (CHASE), a lightweight mechanism that detects modality conflicts and performs inference-time attention steering.
  arXiv  Detail & Related papers  (2026-05-01T18:35:49Z)
• SURE: Synergistic Uncertainty-aware Reasoning for Multimodal Emotion Recognition in Conversations [15.80542007124167]
  Multimodal emotion recognition in conversations (MERC) requires integrating multimodal signals while being robust to noise and modeling contextual reasoning.<n>We propose SURE (Synergistic Uncertainty-aware REasoning) for MERC, a framework that improves robustness and contextual modeling.
  arXiv  Detail & Related papers  (2026-04-02T11:32:15Z)
• Complementarity-Supervised Spectral-Band Routing for Multimodal Emotion Recognition [60.20529806857076]
  Multimodal emotion recognition fuses cues such as text, video, and audio to understand individual emotional states.<n>Prior methods face two main limitations: mechanically relying on independent unimodal performance, and coarse-grained fusion conflicting with the fine-grained representations required by emotion tasks.<n>We propose the Complementarity-Supervised Multi-Band Expert Network, named Atsuko, to model fine-grained complementary features via multi-scale band decomposition and expert collaboration.
  arXiv  Detail & Related papers  (2026-03-07T03:58:48Z)
• Explaining multimodal LLMs via intra-modal token interactions [55.27436637894534]
  Multimodal Large Language Models (MLLMs) have achieved remarkable success across diverse vision-language tasks, yet their internal decision-making mechanisms remain insufficiently understood.<n>We propose enhancing interpretability by leveraging intra-modal interaction.
  arXiv  Detail & Related papers  (2025-09-26T14:39:13Z)
• Latent Distribution Decoupling: A Probabilistic Framework for Uncertainty-Aware Multimodal Emotion Recognition [7.25361375272096]
  Multimodal multi-label emotion recognition aims to identify the concurrent presence of multiple emotions in multimodal data.<n>Existing studies overlook the impact of textbfaleatoric uncertainty, which is the inherent noise in the multimodal data.<n>This paper proposes Latent emotional Distribution Decomposition with Uncertainty perception framework.
  arXiv  Detail & Related papers  (2025-02-19T18:53:23Z)
• Cross-Attention is Not Enough: Incongruity-Aware Dynamic Hierarchical Fusion for Multimodal Affect Recognition [69.32305810128994]
  Incongruity between modalities poses a challenge for multimodal fusion, especially in affect recognition. We propose the Hierarchical Crossmodal Transformer with Dynamic Modality Gating (HCT-DMG), a lightweight incongruity-aware model. HCT-DMG: 1) outperforms previous multimodal models with a reduced size of approximately 0.8M parameters; 2) recognizes hard samples where incongruity makes affect recognition difficult; 3) mitigates the incongruity at the latent level in crossmodal attention.
  arXiv  Detail & Related papers  (2023-05-23T01:24:15Z)
• Exploiting modality-invariant feature for robust multimodal emotion recognition with missing modalities [76.08541852988536]
  We propose to use invariant features for a missing modality imagination network (IF-MMIN) We show that the proposed model outperforms all baselines and invariantly improves the overall emotion recognition performance under uncertain missing-modality conditions.
  arXiv  Detail & Related papers  (2022-10-27T12:16:25Z)
• COLD Fusion: Calibrated and Ordinal Latent Distribution Fusion for Uncertainty-Aware Multimodal Emotion Recognition [14.963637194500029]
  This paper introduces an uncertainty-aware audiovisual fusion approach that quantifies modality-wise uncertainty towards emotion prediction. We impose Ordinal Ranking constraints on the variance vectors of audiovisual latent distributions. Our evaluation on two emotion recognition corpora, AVEC 2019 CES and IEMOCAP, shows that audiovisual emotion recognition can considerably benefit from well-calibrated and well-ranked latent uncertainty measures.
  arXiv  Detail & Related papers  (2022-06-12T20:25:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.