Rationale-Guided Learning for Multimodal Emotion Recognition

• URL: http://arxiv.org/abs/2608.10448v2
• Date: Tue, 18 Aug 2026 03:30:46 GMT
• Title: Rationale-Guided Learning for Multimodal Emotion Recognition
• Abstract summary: Multimodal emotion recognition in conversation requires understanding complex interactions between verbal and non-verbal cues.<n>We propose rationale-guided learning (RGL), a novel framework that transforms MERC into a cognitively-inspired reasoning task.
• Score: 21.87364319362605
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multimodal emotion recognition in conversation (MERC) requires understanding complex interactions between verbal and non-verbal cues. However, most existing approaches fundamentally treat this as a direct input-output (multimodal cues-emotion labels) mapping problem, overlooking the causal reasoning that humans use when interpreting emotions. We propose rationale-guided learning (RGL), a novel framework that transforms MERC into a cognitively-inspired reasoning task. Based on dual-process theory, we decompose emotional reasoning into three facets: Intuitive (immediate perception, System 1), Contextual (situational analysis, System 2), and Integrative (synthesis of both). We leverage an MLLM offline to generate structured rationales, which are encoded as memories to guide model training via aligning internal representations with human-like reasoning patterns. Our final model operates without any MLLM overheads at inference time. Experimental results show that RGL achieves state-of-the-art performance on the IEMOCAP and MELD benchmarks. Further, for interpretation, we demonstrate that the model's internal features effectively retrieve semantically correct rationales for unseen test samples, validating its rationale reasoning capabilities.

Related papers

• Look on Demand: A Cognitive Scheduling Framework for Visual Evidence Acquisition in Multimodal Reasoning [82.86343313807158]
  We argue that a central challenge is how and when visual evidence is introduced into the reasoning process.<n>Motivated by this insight, we propose CSMR, a multimodal reasoning framework in which a language model controls the reasoning process.
  arXiv  Detail & Related papers  (2026-05-27T08:43:13Z)
• EMO-R3: Reflective Reinforcement Learning for Emotional Reasoning in Multimodal Large Language Models [62.3977734456669]
  We propose Reflective Reinforcement Learning for Emotional Reasoning (EMO-R3), a framework designed to enhance the emotional reasoning ability of Multimodal Large Language Models (MLLMs)<n>We introduce Structured Emotional Thinking to guide the model to perform step-by-step emotional reasoning in a structured and interpretable manner, and design a Reflective Emotional Reward that enables the model to re-evaluate its reasoning based on visual-text consistency and emotional coherence.<n>EMO-R3 significantly improves both the interpretability and emotional intelligence of MLLMs, achieving superior performance across multiple visual emotional understanding benchmarks.
  arXiv  Detail & Related papers  (2026-02-27T08:42:52Z)
• Unveiling the Cognitive Compass: Theory-of-Mind-Guided Multimodal Emotion Reasoning [31.790359663851305]
  genuine affective intelligence requires explicit modeling of Theory of Mind (ToM), the cognitive substrate from which emotions arise.<n>We introduce HitEmotion, a ToM-grounded hierarchical benchmark that diagnoses capability breakpoints across increasing levels of cognitive depth.<n>Second, we propose a ToM-guided reasoning chain that tracks mental states and calibrates cross-modal evidence to achieve faithful emotional reasoning.
  arXiv  Detail & Related papers  (2026-02-01T02:26:12Z)
• E^2-LLM: Bridging Neural Signals and Interpretable Affective Analysis [54.763420895859035]
  We present ELLM2-EEG-to-Emotion Large Language Model, first MLLM framework for interpretable emotion analysis from EEG.<n>ELLM integrates a pretrained EEG encoder with Q-based LLMs through learnable projection layers, employing a multi-stage training pipeline.<n>Experiments on the dataset across seven emotion categories demonstrate that ELLM2-EEG-to-Emotion Large Language Model achieves excellent performance on emotion classification.
  arXiv  Detail & Related papers  (2026-01-11T13:21:20Z)
• A Unified Spoken Language Model with Injected Emotional-Attribution Thinking for Human-like Interaction [50.05919688888947]
  This paper presents a unified spoken language model for emotional intelligence, enhanced by a novel data construction strategy termed Injected Emotional-Attribution Thinking (IEAT)<n>IEAT incorporates user emotional states and their underlying causes into the model's internal reasoning process, enabling emotion-aware reasoning to be internalized rather than treated as explicit supervision.<n> Experiments on the Human-like Spoken Dialogue Systems Challenge (HumDial) Emotional Intelligence benchmark demonstrate that the proposed approach achieves top-ranked performance across emotional trajectory modeling, emotional reasoning, and empathetic response generation.
  arXiv  Detail & Related papers  (2026-01-08T14:07:30Z)
• Do LLMs Feel? Teaching Emotion Recognition with Prompts, Retrieval, and Curriculum Learning [16.195689085967004]
  Emotion Recognition in Conversation (ERC) is a crucial task for understanding human emotions and enabling natural human-computer interaction.<n>We propose a novel ERC training framework, PRC-Emo, which integrates Prompt engineering, demonstration Retrieval, and Curriculum learning.<n>We show that our method achieves new state-of-the-art (SOTA) performance, demonstrating the effectiveness and generalizability of our approach.
  arXiv  Detail & Related papers  (2025-11-10T12:52:11Z)
• Emotion-Coherent Reasoning for Multimodal LLMs via Emotional Rationale Verifier [53.55996102181836]
  We propose the Emotional Rationale Verifier (ERV) and an Explanation Reward.<n>Our method guides the model to produce reasoning that is explicitly consistent with the target emotion.<n>We show that our approach not only enhances alignment between explanation and prediction but also empowers MLLMs to deliver emotionally coherent, trustworthy interactions.
  arXiv  Detail & Related papers  (2025-10-27T16:40:17Z)
• Fine-Grained Emotion Recognition via In-Context Learning [24.79387634284384]
  Fine-grained emotion recognition aims to identify the emotional type in queries through reasoning and decision-making processes.<n>Recent methods use In-Context Learning (ICL), enhancing the representation of queries through semantically similar examples.<n>This paper investigates decision-making in fine-grained emotion recognition through prototype theory.
  arXiv  Detail & Related papers  (2025-10-08T03:17:09Z)
• Beyond Classification: Towards Speech Emotion Reasoning with Multitask AudioLLMs [47.325269852330884]
  We develop a strategy to enhance emotion recognition by producing semantically aligned, evidence-grounded explanations.<n>We introduce a unified framework combining reasoning-augmented data supervision, dual-encoder architecture, and task-alternating training.<n> Experiments on IEMOCAP and MELD show that our approach not only improves emotion prediction accuracy but also enhances the coherence and evidential grounding of the generated responses.
  arXiv  Detail & Related papers  (2025-06-07T14:52:58Z)
• DialogueCRN: Contextual Reasoning Networks for Emotion Recognition in Conversations [0.0]
  We propose novel Contextual Reasoning Networks (DialogueCRN) to fully understand the conversational context from a cognitive perspective. Inspired by the Cognitive Theory of Emotion, we design multi-turn reasoning modules to extract and integrate emotional clues. The reasoning module iteratively performs an intuitive retrieving process and a conscious reasoning process, which imitates human unique cognitive thinking.
  arXiv  Detail & Related papers  (2021-06-03T16:47:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.