Leveraging Self-Paced Curriculum Learning for Enhanced Modality Balance in Multimodal Conversational Emotion Recognition

• URL: http://arxiv.org/abs/2605.21565v1
• Date: Wed, 20 May 2026 17:07:16 GMT
• Title: Leveraging Self-Paced Curriculum Learning for Enhanced Modality Balance in Multimodal Conversational Emotion Recognition
• Abstract summary: Multimodal Emotion Recognition in Conversations (MERC) is a crucial task for understanding human interactions.<n>We propose a plug-and-play framework based on Self-Paced Curriculum Learning (SPCL) for MERC.<n>We introduce a dual-level Difficulty Measurer that captures both utterance-level and conversation-level challenges.
• Score: 2.458499866376012
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multimodal Emotion Recognition in Conversations (MERC) is a crucial task for understanding human interactions, where multimodal approaches integrating language, facial expressions, and vocal tone have achieved significant progress. However, modality misalignment and imbalanced learning remain major challenges, limiting the effective utilization of multimodal information. To address this issue, we propose a plug-and-play framework based on Self-Paced Curriculum Learning (SPCL) for MERC. We introduce a dual-level Difficulty Measurer that captures both utterance-level and conversation-level challenges. The utterance-level score models fine-grained modality-specific difficulty, while the conversation-level score captures broader dialogue structures, including emotional dependencies and modality coherence. Based on these scores, the Learning Scheduler dynamically guides training from easier to more difficult instances. By integrating SPCL into existing MERC architectures, our method alleviates modality imbalance and improves model robustness. Extensive experiments on the IEMOCAP and MELD datasets demonstrate consistent improvements across different architectures and modality settings. On IEMOCAP, SPCL improves weighted F1-score by approximately +1.2% to +6.6% over baseline models, while on MELD, gains reach up to +10.4%. These results highlight the effectiveness and generalizability of SPCL as a lightweight plug-and-play module for multimodal emotion recognition.

Related papers

• EII-SCL: Harnessing Emotional Inertia for Multimodal Emotion Recognition in Conversation [51.04910642896281]
  Multimodal emotion recognition in conversation (MERC) achieves accurate predictions by integrating multimodal and contextual information in dialogues.<n>Current MERC approaches overlook the impact of contextual emotional inertia in emotion shift, leading to sub-optimal performance.<n>We propose a novel Emotional Inertia-Informed Supervised Contrastive Learning module (EII-SCL) that informs the contrastive objective by constructing inertia-affected samples within temporal windows.
  arXiv  Detail & Related papers  (2026-07-19T18:09:54Z)
• SPARD: Self-Paced Curriculum for RL Alignment via Integrating Reward Dynamics and Data Utility [71.76390626651254]
  We propose SPARD, a framework that establishes an automated, self-paced curriculum by perceiving learning progress to adjust dynamically multi-objective reward weights and data importance.<n>Extensive experiments across multiple benchmarks demonstrate that SPARD significantly enhances model capabilities across all domains.
  arXiv  Detail & Related papers  (2026-04-09T05:37:22Z)
• Label What Matters: Modality-Balanced and Difficulty-Aware Multimodal Active Learning [13.136629024689432]
  We propose RL-MBA, a reinforcement-learning framework for modality-balanced, difficulty-aware active learning.<n>Experiments on Food101, KineticsSound, and VGGSound demonstrate that RL-MBA consistently outperforms strong baselines.
  arXiv  Detail & Related papers  (2026-03-26T07:26:48Z)
• UniICL: Systematizing Unified Multimodal In-context Learning through a Capability-Oriented Taxonomy [99.95663439922937]
  In-context Learning efficacy is often non-monotonic and highly task-dependent.<n>We introduce a six-level capability-oriented taxonomy that categorizes the functional role of demonstrations.<n>We construct UniICL-760K, a large-scale corpus featuring curated 8-shot In-context Learning episodes.<n>As an architectural intervention, we propose the Context-Adaptive Prototype Modulator.
  arXiv  Detail & Related papers  (2026-03-25T18:09:33Z)
• Optimizing Conversational Quality in Spoken Dialogue Systems with Reinforcement Learning from AI Feedback [82.70507055599093]
  We present the first systematic study of preference learning for improving SDS quality in both multi-turn Chain-of-Thought and blockwise duplex models.<n>Experiments show that single-reward RLAIF selectively improves its targeted metric, while joint multi-reward training yields consistent gains across semantic quality and audio naturalness.
  arXiv  Detail & Related papers  (2026-01-27T00:55:14Z)
• Multi-Loss Learning for Speech Emotion Recognition with Energy-Adaptive Mixup and Frame-Level Attention [27.15999842662482]
  Speech emotion recognition (SER) is an important technology in human-computer interaction.<n>We propose a multi-loss learning framework integrating an energy-adaptive mixup (EAM) method and a frame-level attention module (FLAM)<n>We evaluate our method on four widely used SER datasets: IEMOCAP, MSP-IMPROV, RAVDESS, and SAVEE.
  arXiv  Detail & Related papers  (2025-12-04T08:04:45Z)
• MoIIE: Mixture of Intra- and Inter-Modality Experts for Large Vision Language Models [52.876185634349575]
  We propose to incorporate Mixture of Intra- and Inter-Modality Experts (MoIIE) to Large Vision-Language Models (LVLMs)<n>For each token, expert routing is guided by its modality, directing tokens to their respective intra-modality experts as well as a shared pool of inter-modality experts.<n>Our MoIIE models with 5.5B and 11.3B activated parameters match or even surpass the performance of existing advanced open-source MoE-LLMs based multi-modal models.
  arXiv  Detail & Related papers  (2025-08-13T13:00:05Z)
• PUMA: Layer-Pruned Language Model for Efficient Unified Multimodal Retrieval with Modality-Adaptive Learning [54.73049408950049]
  We propose a Layer-Pruned Language Model for Efficient Unified Multimodal Retrieval with Modality-Adaptive Learning.<n>Our approach improves unified multimodal retrieval from both structural and learning perspectives.
  arXiv  Detail & Related papers  (2025-07-10T16:47:25Z)
• DynCIM: Dynamic Curriculum for Imbalanced Multimodal Learning [27.20479303843989]
  DynCIM is a novel dynamic curriculum learning framework designed to quantify the inherent imbalances from both sample and modality perspectives.<n>DynCIM employs a sample-level curriculum to dynamically assess each sample's difficulty according to prediction deviation, consistency, and stability.<n>A modality-level curriculum measures modality contributions from global and local.
  arXiv  Detail & Related papers  (2025-03-09T05:30:15Z)
• Knowledge-Decoupled Synergetic Learning: An MLLM based Collaborative Approach to Few-shot Multimodal Dialogue Intention Recognition [17.790383360652704]
  Training for few-shot multimodal dialogue intention recognition involves two interconnected tasks.<n>This phenomenon is attributed to knowledge interference stemming from the superposition of weight matrix updates during the training process.<n>We propose Knowledge-Decoupled Synergetic Learning, which transforms knowledge into interpretable rules, while applying the post-training of larger models.
  arXiv  Detail & Related papers  (2025-03-06T08:28:44Z)
• Curriculum Learning Meets Directed Acyclic Graph for Multimodal Emotion Recognition [2.4660652494309936]
  MultiDAG+CL is a novel approach for Multimodal Emotion Recognition in Conversation (ERC) The model is enhanced by Curriculum Learning (CL) to address challenges related to emotional shifts and data imbalance. Experimental results on the IEMOCAP and MELD datasets demonstrate that the MultiDAG+CL models outperform baseline models.
  arXiv  Detail & Related papers  (2024-02-27T07:28:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.