Multimodal Emotion Regression with Multi-Objective Optimization and VAD-Aware Audio Modeling for the 10th ABAW EMI Track

• URL: http://arxiv.org/abs/2603.13760v1
• Date: Sat, 14 Mar 2026 05:17:06 GMT
• Title: Multimodal Emotion Regression with Multi-Objective Optimization and VAD-Aware Audio Modeling for the 10th ABAW EMI Track
• Abstract summary: This paper focuses on the Emotional Mimicry Intensity (EMI) Estimation track on the Hume-Vidmimic2 dataset.<n>It aims to predict six continuous emotion dimensions: Admiration, Amusement, Determination, Empathic Pain, Excitement, and Joy.<n>Our framework integrates concatenation-based multimodal fusion, a shared six-dimensional regression head, multi-objective optimization with MSE, Pearson-correlation, and auxiliary branch supervision, EMA for parameter stabilization, and a VAD-inspired latent prior for the acoustic branch.
• Score: 11.775637010038794
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We participated in the 10th ABAW Challenge, focusing on the Emotional Mimicry Intensity (EMI) Estimation track on the Hume-Vidmimic2 dataset. This task aims to predict six continuous emotion dimensions: Admiration, Amusement, Determination, Empathic Pain, Excitement, and Joy. Through systematic multimodal exploration of pretrained high-level features, we found that, under our pretrained feature setting, direct feature concatenation outperformed the more complex fusion strategies we tested. This empirical finding motivated us to design a systematic approach built upon three core principles: (i) preserving modality-specific attributes through feature-level concatenation; (ii) improving training stability and metric alignment via multi-objective optimization; and (iii) enriching acoustic representations with a VAD-inspired latent prior. Our final framework integrates concatenation-based multimodal fusion, a shared six-dimensional regression head, multi-objective optimization with MSE, Pearson-correlation, and auxiliary branch supervision, EMA for parameter stabilization, and a VAD-inspired latent prior for the acoustic branch. On the official validation set, the proposed scheme achieved our best mean Pearson Correlation Coefficient of 0.478567.

Related papers

• Enhancing Multimodal Emotion Recognition via Multi-Feature Encoding and Attention-Based Fusion [5.740228163421595]
  This paper proposes a novel multimodal emotion recognition framework.<n>It integrates rich audio and visual feature extraction with an attention-based fusion strategy.<n>Our findings suggest that the proposed framework effectively captures diverse emotional cues from speech and visual expressions.
  arXiv  Detail & Related papers  (2026-09-04T03:40:16Z)
• Traits Run Deeper: Trait-Specific Asymmetric Fusion for Personality Assessment [53.50824432699408]
  Traits Run Deeper is a novel personality assessment framework.<n>It consists of three components: the Multimodal Foundation Representation (MFR), the Trait-Specific Modality Fusion (TSMF) and the Distribution-Calibrated Personality Regression (DCPR)
  arXiv  Detail & Related papers  (2026-06-09T06:38:36Z)
• Two-Stage Multimodal Framework for Emotion Mimicry Intensity Prediction [16.051484505286783]
  We present our submission to the Hume-ABAW10 Emotional Mimicry Intensity (EMI) Challenge.<n>We propose a staged multimodal framework that combines textual, acoustic, and visual representations, with an optional motion branch.<n>Our team was placed third in the EMI challenge, achieving an average Pearson correlation of 0.57 for the test set.
  arXiv  Detail & Related papers  (2026-05-21T01:29:26Z)
• Plug, Play, and Fortify: A Low-Cost Module for Robust Multimodal Image Understanding Models [6.350443894942629]
  Multimodal Weight Allocation Module (MWAM) is a plug-and-play component that dynamically re-balances the contribution of each branch during training.<n>MWAM delivers consistent performance gains across a wide range of tasks and modality combinations.
  arXiv  Detail & Related papers  (2026-02-26T05:51:41Z)
• Improving Multimodal Sentiment Analysis via Modality Optimization and Dynamic Primary Modality Selection [54.10252086842123]
  Multimodal Sentiment Analysis (MSA) aims to predict sentiment from language, acoustic, and visual data in videos.<n>This paper proposes a modality optimization and dynamic primary modality selection framework (MODS)<n>Experiments on four benchmark datasets demonstrate that MODS outperforms state-of-the-art methods.
  arXiv  Detail & Related papers  (2025-11-09T11:13:32Z)
• Dynamic Fusion Multimodal Network for SpeechWellness Detection [7.169178956727836]
  Suicide is one of the leading causes of death among adolescents.<n>Previous suicide risk prediction studies have primarily focused on either textual or acoustic information in isolation.<n>We explore a lightweight multi-branch multimodal system based on a dynamic fusion mechanism for speech detection.
  arXiv  Detail & Related papers  (2025-08-25T14:18:12Z)
• WSM: Decay-Free Learning Rate Schedule via Checkpoint Merging for LLM Pre-training [64.0932926819307]
  We present Warmup-Stable and Merge (WSM), a framework that establishes a formal connection between learning rate decay and model merging.<n>WSM provides a unified theoretical foundation for emulating various decay strategies.<n>Our framework consistently outperforms the widely-adopted Warmup-Stable-Decay (WSD) approach across multiple benchmarks.
  arXiv  Detail & Related papers  (2025-07-23T16:02:06Z)
• Multi-Level Aware Preference Learning: Enhancing RLHF for Complex Multi-Instruction Tasks [81.44256822500257]
  RLHF has emerged as a predominant approach for aligning artificial intelligence systems with human preferences.<n> RLHF exhibits insufficient compliance capabilities when confronted with complex multi-instruction tasks.<n>We propose a novel Multi-level Aware Preference Learning (MAPL) framework, capable of enhancing multi-instruction capabilities.
  arXiv  Detail & Related papers  (2025-05-19T08:33:11Z)
• Will Pre-Training Ever End? A First Step Toward Next-Generation Foundation MLLMs via Self-Improving Systematic Cognition [89.50068130832635]
  Self-Improving cognition (SIcog) is a self-learning framework for constructing next-generation foundation MLLMs by multimodal knowledge.<n>We propose Chain-of-Description for step-by-step visual understanding and integrate structured Chain-of-Thought (CoT) reasoning to support in-depth multimodal reasoning.<n>Experiments demonstrate SIcog's effectiveness in developing MLLMs with enhanced multimodal cognition.
  arXiv  Detail & Related papers  (2025-03-16T00:25:13Z)
• Technical Approach for the EMI Challenge in the 8th Affective Behavior Analysis in-the-Wild Competition [10.741278852581646]
  Emotional Mimicry Intensity (EMI) estimation plays a pivotal role in understanding human social behavior and advancing human-computer interaction.<n>This paper proposes a dual-stage cross-modal alignment framework to address the limitations of existing methods.<n> Experiments on the Hume-Vidmimic2 dataset demonstrate superior performance with an average Pearson coefficient correlation of 0.51 across six emotion dimensions.
  arXiv  Detail & Related papers  (2025-03-13T17:46:16Z)
• Qieemo: Speech Is All You Need in the Emotion Recognition in Conversations [1.0690007351232649]
  Multimodal approaches benefit from the fusion of diverse modalities, thereby improving the recognition accuracy.<n>The proposed Qieemo framework effectively utilizes the pretrained automatic speech recognition (ASR) model which contains naturally frame aligned textual and emotional features.<n>The experimental results on the IEMOCAP dataset demonstrate that Qieemo outperforms the benchmark unimodal, multimodal, and self-supervised models with absolute improvements of 3.0%, 1.2%, and 1.9% respectively.
  arXiv  Detail & Related papers  (2025-03-05T07:02:30Z)
• Baichuan-Omni-1.5 Technical Report [78.49101296394218]
  Baichuan- Omni-1.5 is an omni-modal model that not only has omni-modal understanding capabilities but also provides end-to-end audio generation capabilities.<n>We establish a comprehensive data cleaning and synthesis pipeline for multimodal data, obtaining about 500B high-quality data.<n>Second, an audio-tokenizer has been designed to capture both semantic and acoustic information from audio, enabling seamless integration and enhanced compatibility with MLLM.
  arXiv  Detail & Related papers  (2025-01-26T02:19:03Z)
• T-REX: Mixture-of-Rank-One-Experts with Semantic-aware Intuition for Multi-task Large Language Model Finetuning [31.276142111455847]
  Large language models (LLMs) encounter significant adaptation challenges in diverse multitask finetuning.<n>We design a novel framework, mixunderlinetextbfTureunderlinetextbf-of-underlinetextbfRank-onunderlinetextbfE-eunderlinetextbfXper ts (textttT-REX)<n>Rank-1 experts enable a mix-and-match mechanism to quadratically expand the vector subspace of experts with linear parameter overheads, achieving approximate error reduction with optimal
  arXiv  Detail & Related papers  (2024-04-13T12:14:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.