EmoMM: Benchmarking and Steering MLLM for Multimodal Emotion Recognition under Conflict and Missingness

• URL: http://arxiv.org/abs/2605.01024v1
• Date: Fri, 01 May 2026 18:35:49 GMT
• Title: EmoMM: Benchmarking and Steering MLLM for Multimodal Emotion Recognition under Conflict and Missingness
• Abstract summary: We introduce EmoMM, a comprehensive benchmark featuring modality-aligned, conflict, and missing subsets.<n>We propose Conflict-aware Head-level Attention Steering (CHASE), a lightweight mechanism that detects modality conflicts and performs inference-time attention steering.
• Score: 17.292747703126796
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multimodal Emotion Recognition (MER) is critical for interpreting real-world interactions. While Multimodal Large Language Models (MLLM) have shown promise in MER, their internal decision-making mechanisms under modality conflict and missingness remain largely underexplored. In this paper, to systematically investigate these behaviors, we introduce EmoMM, a comprehensive benchmark featuring modality-aligned, conflict, and missing subsets. Through extensive evaluation, we uncover a Video Contribution Collapse (VCC) phenomenon, where MLLM marginalize video evidence due to high token redundancy and modality preferences. To address this, we propose Conflict-aware Head-level Attention Steering (CHASE), a lightweight mechanism that detects modality conflicts and performs inference-time attention steering, effectively mitigating decision bias without retraining the backbone. Experimental results demonstrate that CHASE consistently improves performance across various settings, significantly enhancing the reliability of MLLM in complex affective scenarios.

Related papers

• EmoEUS: Uncertainty Supervision for Multimodal Emotion Recognition in Conversation [58.055014746373054]
  We propose EmoEUS, an explicit uncertainty supervision framework for multimodal emotion recognition.<n>EmoEUS performs uncertainty-aware multimodal fusion by dynamically weighting modalities using learned variance estimates.<n>Experiments on IEMOCAP and MELD show that EmoEUS consistently outperforms state-of-the-art methods.
  arXiv  Detail & Related papers  (2026-07-19T18:09:22Z)
• Progressive Representation Learning for Multimodal Sentiment Analysis with Incomplete Modalities [26.613566093134065]
  Multimodal Sentiment Analysis seeks to infer human emotions by integrating textual, acoustic, and visual cues.<n>Existing approaches often rely on all modalities are completeness, whereas real-world applications frequently encounter noise, hardware failures, or privacy restrictions that result in missing modalities.<n>We propose PRLF, a Progressive Representation Learning Framework designed for MSA under uncertain missing-modality conditions.
  arXiv  Detail & Related papers  (2026-03-10T02:45:02Z)
• Beyond Unimodal Shortcuts: MLLMs as Cross-Modal Reasoners for Grounded Named Entity Recognition [51.68340973140949]
  Multimodal Named Entity Recognition (GMNER) aims to extract text-based entities, assign them semantic categories, and ground them to corresponding visual regions.<n> MLLMs exhibit $textbfmodality bias$, including visual bias and textual bias, which stems from their tendency to take unimodal shortcuts.<n>We propose Modality-aware Consistency Reasoning ($bfMCR$), which enforces structured cross-modal reasoning.
  arXiv  Detail & Related papers  (2026-02-04T12:12:49Z)
• Explaining multimodal LLMs via intra-modal token interactions [55.27436637894534]
  Multimodal Large Language Models (MLLMs) have achieved remarkable success across diverse vision-language tasks, yet their internal decision-making mechanisms remain insufficiently understood.<n>We propose enhancing interpretability by leveraging intra-modal interaction.
  arXiv  Detail & Related papers  (2025-09-26T14:39:13Z)
• Benchmarking and Bridging Emotion Conflicts for Multimodal Emotion Reasoning [21.344503400857107]
  We introduce CA-MER, a new benchmark designed to examine MLLMs under realistic emotion conflicts.<n> evaluations reveal that current state-of-the-art emotion MLLMs systematically over-rely on audio signal during emotion conflicts.<n>We propose MoSEAR, a parameter-efficient framework that promotes balanced modality integration.
  arXiv  Detail & Related papers  (2025-08-02T04:03:44Z)
• Multimodal Video Emotion Recognition with Reliable Reasoning Priors [24.692842008018545]
  We employ Gemini to generate fine-grained, modality-separable reasoning traces, which are injected as priors during the fusion stage.<n>We introduce Balanced Dual-Contrastive Learning, a loss formulation that jointly balances inter-class and intra-class distributions.
  arXiv  Detail & Related papers  (2025-07-29T15:55:23Z)
• Robust Multimodal Large Language Models Against Modality Conflict [94.12341487880465]
  multimodal large language models (MLLMs) are prone to hallucinations in real-world scenarios.<n>We study the inherent conflicts in inputs from different modalities that place MLLMs in a dilemma and directly lead to hallucinations.<n>Three methods are proposed to alleviate the hallucination caused by modality conflict.
  arXiv  Detail & Related papers  (2025-07-09T11:18:38Z)
• Survey of Adversarial Robustness in Multimodal Large Language Models [17.926240920647892]
  Multimodal Large Language Models (MLLMs) have demonstrated exceptional performance in artificial intelligence.<n>Their deployment in real-world applications raises significant concerns about adversarial vulnerabilities.<n>This paper reviews the adversarial robustness of MLLMs, covering different modalities.
  arXiv  Detail & Related papers  (2025-03-18T06:54:59Z)
• Benchmarking Gaslighting Negation Attacks Against Multimodal Large Language Models [45.63440666848143]
  Multimodal Large Language Models (MLLMs) have exhibited remarkable advancements in integrating different modalities.<n>Despite their success, MLLMs remain vulnerable to conversational adversarial inputs.<n>We study gaslighting negation attacks: a phenomenon where models, despite initially providing correct answers, are persuaded by user-provided negations to reverse their outputs.
  arXiv  Detail & Related papers  (2025-01-31T10:37:48Z)
• Insight Over Sight: Exploring the Vision-Knowledge Conflicts in Multimodal LLMs [55.74117540987519]
  This paper explores the problem of commonsense level vision-knowledge conflict in Multimodal Large Language Models (MLLMs)<n>We introduce an automated framework, augmented with human-in-the-loop quality control, to generate inputs designed to simulate and evaluate these conflicts in MLLMs.<n>Using this framework, we have crafted a diagnostic benchmark consisting of 374 original images and 1,122 high-quality question-answer pairs.
  arXiv  Detail & Related papers  (2024-10-10T17:31:17Z)
• Cross-Attention is Not Enough: Incongruity-Aware Dynamic Hierarchical Fusion for Multimodal Affect Recognition [69.32305810128994]
  Incongruity between modalities poses a challenge for multimodal fusion, especially in affect recognition. We propose the Hierarchical Crossmodal Transformer with Dynamic Modality Gating (HCT-DMG), a lightweight incongruity-aware model. HCT-DMG: 1) outperforms previous multimodal models with a reduced size of approximately 0.8M parameters; 2) recognizes hard samples where incongruity makes affect recognition difficult; 3) mitigates the incongruity at the latent level in crossmodal attention.
  arXiv  Detail & Related papers  (2023-05-23T01:24:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.