Two-Stage Multimodal Framework for Emotion Mimicry Intensity Prediction

• URL: http://arxiv.org/abs/2605.21869v1
• Date: Thu, 21 May 2026 01:29:26 GMT
• Title: Two-Stage Multimodal Framework for Emotion Mimicry Intensity Prediction
• Abstract summary: We present our submission to the Hume-ABAW10 Emotional Mimicry Intensity (EMI) Challenge.<n>We propose a staged multimodal framework that combines textual, acoustic, and visual representations, with an optional motion branch.<n>Our team was placed third in the EMI challenge, achieving an average Pearson correlation of 0.57 for the test set.
• Score: 16.051484505286783
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present our submission to the Hume-ABAW10 Emotional Mimicry Intensity (EMI) Challenge, which aims to predict six continuous emotion intensity dimensions: Admiration, Amusement, Determination, Empathic Pain, Excitement, and Joy, from in-the-wild multimodal video clips. We propose a staged multimodal framework that combines textual, acoustic, and visual representations, with an optional motion branch. Our approach first trains modality-specific encoders independently and then fuses their learned representations through a lightweight regressor with modality dropout and controlled encoder adaptation. Across our submitted systems, the best validation performance is obtained by the text--audio--vision--motion fusion model under the expanded 4:1 split, achieving an average Pearson correlation of 0.4722. Although the motion branch yields only very slight gains, its behavior can be interesting to study. Our team was placed third in the EMI challenge, achieving an average Pearson correlation of 0.57 for the test set. Overall, we provide a practical and reproducible baseline for EMI prediction.

Related papers

• P2Fusion: Prompt-based Progressive Infrared-Visible Image Fusion via Dual-Prior Distillation [73.147281449267]
  We propose P2Fusion, a prior-guided distillation-based framework that reformulates infrared-visible image fusion.<n>Instead of imposing hard-coded penalties, we distill image-intrinsic priors, thermal saliency and spatial quality, into learnable dynamic regulators.<n>P2Fusion achieves state-of-the-art results in 14 out of 20 key evaluation metrics across 5 benchmarks.
  arXiv  Detail & Related papers  (2026-08-13T10:09:49Z)
• Multimodal Emotion Regression with Multi-Objective Optimization and VAD-Aware Audio Modeling for the 10th ABAW EMI Track [11.775637010038794]
  This paper focuses on the Emotional Mimicry Intensity (EMI) Estimation track on the Hume-Vidmimic2 dataset.<n>It aims to predict six continuous emotion dimensions: Admiration, Amusement, Determination, Empathic Pain, Excitement, and Joy.<n>Our framework integrates concatenation-based multimodal fusion, a shared six-dimensional regression head, multi-objective optimization with MSE, Pearson-correlation, and auxiliary branch supervision, EMA for parameter stabilization, and a VAD-inspired latent prior for the acoustic branch.
  arXiv  Detail & Related papers  (2026-03-14T05:17:06Z)
• Team RAS in 10th ABAW Competition: Multimodal Valence and Arousal Estimation Approach [53.64487844936037]
  Our method combines three complementary modalities: face, behavior, and audio.<n>Experiments demonstrate that the proposed multimodal fusion strategy achieves a Concordance Correlation Coefficient (CCC) of 0.658 on the Aff-Wild2 development set.
  arXiv  Detail & Related papers  (2026-03-13T15:06:14Z)
• Team LEYA in 10th ABAW Competition: Multimodal Ambivalence/Hesitancy Recognition Approach [43.09390014734485]
  This paper presents a multimodal approach for ambivalence/hesitancy recognition in unconstrained videos.<n>The proposed approach integrates four complementary modalities: scene, face, audio, and text.<n>Experiments on the BAH corpus demonstrate clear gains of multimodal fusion over all unimodal baselines.
  arXiv  Detail & Related papers  (2026-03-13T09:50:03Z)
• Solution to the 10th ABAW Expression Recognition Challenge: A Robust Multimodal Framework with Safe Cross-Attention and Modality Dropout [20.96260280449496]
  We propose a multimodal framework that dynamically fuses visual and audio representations.<n>Our approach uses a dual-branch Transformer architecture featuring a safe cross-attention mechanism and a modality dropout strategy.<n>This design allows the network to rely on audio-based predictions when visual cues are absent.
  arXiv  Detail & Related papers  (2026-03-09T07:13:20Z)
• Complementarity-Supervised Spectral-Band Routing for Multimodal Emotion Recognition [60.20529806857076]
  Multimodal emotion recognition fuses cues such as text, video, and audio to understand individual emotional states.<n>Prior methods face two main limitations: mechanically relying on independent unimodal performance, and coarse-grained fusion conflicting with the fine-grained representations required by emotion tasks.<n>We propose the Complementarity-Supervised Multi-Band Expert Network, named Atsuko, to model fine-grained complementary features via multi-scale band decomposition and expert collaboration.
  arXiv  Detail & Related papers  (2026-03-07T03:58:48Z)
• Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuning [55.127202990679976]
  We introduce the MERR dataset, containing 28,618 coarse-grained and 4,487 fine-grained annotated samples across diverse emotional categories. This dataset enables models to learn from varied scenarios and generalize to real-world applications. We propose Emotion-LLaMA, a model that seamlessly integrates audio, visual, and textual inputs through emotion-specific encoders.
  arXiv  Detail & Related papers  (2024-06-17T03:01:22Z)
• Efficient Feature Extraction and Late Fusion Strategy for Audiovisual Emotional Mimicry Intensity Estimation [8.529105068848828]
  The Emotional Mimicry Intensity (EMI) Estimation challenge task aims to evaluate the emotional intensity of seed videos. We extracted rich dual-channel visual features based on ResNet18 and AUs for the video modality and effective single-channel features based on Wav2Vec2.0 for the audio modality. We averaged the predictions of the visual and acoustic models, resulting in a more accurate estimation of audiovisual emotional mimicry intensity.
  arXiv  Detail & Related papers  (2024-03-18T13:11:10Z)
• Joint Multimodal Transformer for Emotion Recognition in the Wild [49.735299182004404]
  Multimodal emotion recognition (MMER) systems typically outperform unimodal systems. This paper proposes an MMER method that relies on a joint multimodal transformer (JMT) for fusion with key-based cross-attention.
  arXiv  Detail & Related papers  (2024-03-15T17:23:38Z)
• Cross-Attention is Not Enough: Incongruity-Aware Dynamic Hierarchical Fusion for Multimodal Affect Recognition [69.32305810128994]
  Incongruity between modalities poses a challenge for multimodal fusion, especially in affect recognition. We propose the Hierarchical Crossmodal Transformer with Dynamic Modality Gating (HCT-DMG), a lightweight incongruity-aware model. HCT-DMG: 1) outperforms previous multimodal models with a reduced size of approximately 0.8M parameters; 2) recognizes hard samples where incongruity makes affect recognition difficult; 3) mitigates the incongruity at the latent level in crossmodal attention.
  arXiv  Detail & Related papers  (2023-05-23T01:24:15Z)
• Facial Affect Recognition based on Transformer Encoder and Audiovisual Fusion for the ABAW5 Challenge [10.88275919652131]
  We present our solutions for four sub-challenges of Valence-Arousal (VA) Estimation, Expression (Expr) Classification, Action Unit (AU) Detection and Emotional Reaction Intensity (ERI) Estimation. The 5th ABAW competition focuses on facial affect recognition utilizing different modalities and datasets.
  arXiv  Detail & Related papers  (2023-03-16T08:47:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.