Anchoring Emotions in Text: Robust Multimodal Fusion for Mimicry Intensity Estimation

• URL: http://arxiv.org/abs/2603.14976v1
• Date: Mon, 16 Mar 2026 08:37:37 GMT
• Title: Anchoring Emotions in Text: Robust Multimodal Fusion for Mimicry Intensity Estimation
• Abstract summary: We propose TAEMI (Text-Anchored Emotional Mimicry Intensity estimation), a novel multimodal framework designed for the 10th ABAW Competition.<n>Motivated by the observation that continuous visual and acoustic signals are highly susceptible to transient environmental noise, we break the traditional symmetric fusion paradigm.<n>Our framework achieves a state-of-the-art mean Pearson correlation coefficient across six continuous emotional dimensions, significantly outperforming existing baseline methods.
• Score: 19.86687369529118
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Estimating Emotional Mimicry Intensity (EMI) in naturalistic environments is a critical yet challenging task in affective computing. The primary difficulty lies in effectively modeling the complex, nonlinear temporal dynamics across highly heterogeneous modalities, especially when physical signals are corrupted or missing. To tackle this, we propose TAEMI (Text-Anchored Emotional Mimicry Intensity estimation), a novel multimodal framework designed for the 10th ABAW Competition. Motivated by the observation that continuous visual and acoustic signals are highly susceptible to transient environmental noise, we break the traditional symmetric fusion paradigm. Instead, we leverage textual transcript--which inherently encode a stable, time-independent semantic prior--as central anchors. Specifically, we introduce a Text-Anchored Dual Cross-Attention mechanism that utilizes these robust textual queries to actively filter out frame-level redundancies and align the noisy physical streams. Furthermore, to prevent catastrophic performance degradation caused by inevitably missing data in unconstrained real-world scenarios, we integrate Learnable Missing-Modality Tokens and a Modality Dropout strategy during training. Extensive experiments on the Hume-Vidmimic2 dataset demonstrate that TAEMI effectively captures fine-grained emotional variations and maintains robust predictive resilience under imperfect conditions. Our framework achieves a state-of-the-art mean Pearson correlation coefficient across six continuous emotional dimensions, significantly outperforming existing baseline methods.

Related papers

• EII-SCL: Harnessing Emotional Inertia for Multimodal Emotion Recognition in Conversation [51.04910642896281]
  Multimodal emotion recognition in conversation (MERC) achieves accurate predictions by integrating multimodal and contextual information in dialogues.<n>Current MERC approaches overlook the impact of contextual emotional inertia in emotion shift, leading to sub-optimal performance.<n>We propose a novel Emotional Inertia-Informed Supervised Contrastive Learning module (EII-SCL) that informs the contrastive objective by constructing inertia-affected samples within temporal windows.
  arXiv  Detail & Related papers  (2026-07-19T18:09:54Z)
• UniNDM: A Unified Noise-driven Detection and Mitigation Framework Against Sexual Content in Text-to-Image Generation [90.0896070825457]
  We propose a unified noise-driven framework that rethinks safety mechanisms through the lens of noise dynamics in diffusion processes.<n>Our key insight is that early-stage predicted noise exhibits inherent separability between normal and sexually explicit content.<n>We develop a lightweight noise-based detector achieving superior accuracy with virtually no computational overhead.
  arXiv  Detail & Related papers  (2026-07-18T14:03:39Z)
• Decoding Naturalistic Emotion Dynamics from the Brain: An LLM-Enhanced Regression Framework [3.8543064191088203]
  Decoding emotional states from neural signals has been typically framed as a discrete, single-label classification task.<n>This study reconceptualizes emotion decoding by adopting a multi-target regression framework to track multiple overlapping emotional dimensions as continuous trajectories over time.
  arXiv  Detail & Related papers  (2026-06-05T10:40:22Z)
• Adaptive Evidential Learning for Temporal-Semantic Robustness in Moment Retrieval [39.603000380180774]
  Debiased Evidential Learning for Moment Retrieval (DEMR) is a novel framework that incorporates a Reflective Flipped Fusion (RFF) block for cross-modal alignment.<n>We introduce a Geom-regularizer to refine uncertainty predictions, enabling adaptive alignment with difficult moments and improving retrieval accuracy.
  arXiv  Detail & Related papers  (2025-11-30T16:13:20Z)
• Cross-Modal Consistency-Guided Active Learning for Affective BCI Systems [1.9556470931534158]
  We propose an uncertainty-aware active learning framework that enhances robustness to label noise.<n>Instead of relying solely on EEG-based uncertainty estimates, the method evaluates cross-modal alignment.<n>This feedback-driven process guides the network toward reliable, informative samples and reduces the impact of noisy labels.
  arXiv  Detail & Related papers  (2025-11-19T05:33:48Z)
• Disentangling Emotional Bases and Transient Fluctuations: A Low-Rank Sparse Decomposition Approach for Video Affective Analysis [28.120677578256366]
  Video-based Affective Computing (VAC) suffers from model instability and representational degradation due to complex emotional dynamics.<n>We propose the Low-Rank Sparse Emotion Understanding Framework (LSEF), a unified model grounded in the LowRank Sparse Principle Principle.<n>LSEF employs three plug-and-play modules, which disentangle affective dynamics as a hierarchical low-rank sparse compositional process.
  arXiv  Detail & Related papers  (2025-11-14T15:35:11Z)
• Text-guided Weakly Supervised Framework for Dynamic Facial Expression Recognition [49.41688891301643]
  Dynamic facial expression recognition aims to identify emotional states by modeling the temporal changes in facial movements across video sequences.<n>A key challenge in DFER is the many-to-one labeling problem, where a video composed of numerous frames is assigned a single emotion label.<n>We propose TG-DFER, a text-guided weakly supervised framework that enhances MIL-based DFER by incorporating semantic guidance and coherent temporal modeling.
  arXiv  Detail & Related papers  (2025-11-14T04:49:58Z)
• Investigating Safety Vulnerabilities of Large Audio-Language Models Under Speaker Emotional Variations [94.62792643569567]
  This work systematically investigates the role of speaker emotion.<n>We construct a dataset of malicious speech instructions expressed across multiple emotions and intensities, and evaluate several state-of-the-art LALMs.<n>Our results reveal substantial safety inconsistencies: different emotions elicit varying levels of unsafe responses, and the effect of intensity is non-monotonic, with medium expressions often posing the greatest risk.
  arXiv  Detail & Related papers  (2025-10-19T15:41:25Z)
• DeceptionBench: A Comprehensive Benchmark for AI Deception Behaviors in Real-world Scenarios [57.327907850766785]
  characterization of deception across realistic real-world scenarios remains underexplored.<n>We establish DeceptionBench, the first benchmark that systematically evaluates how deceptive tendencies manifest across different domains.<n>On the intrinsic dimension, we explore whether models exhibit self-interested egoistic tendencies or sycophantic behaviors that prioritize user appeasement.<n>We incorporate sustained multi-turn interaction loops to construct a more realistic simulation of real-world feedback dynamics.
  arXiv  Detail & Related papers  (2025-10-17T10:14:26Z)
• Drift No More? Context Equilibria in Multi-Turn LLM Interactions [58.69551510148673]
  contexts drift is the gradual divergence of a model's outputs from goal-consistent behavior across turns.<n>Unlike single-turn errors, drift unfolds temporally and is poorly captured by static evaluation metrics.<n>We show that multi-turn drift can be understood as a controllable equilibrium phenomenon rather than as inevitable decay.
  arXiv  Detail & Related papers  (2025-10-09T04:48:49Z)
• Technical Approach for the EMI Challenge in the 8th Affective Behavior Analysis in-the-Wild Competition [10.741278852581646]
  Emotional Mimicry Intensity (EMI) estimation plays a pivotal role in understanding human social behavior and advancing human-computer interaction.<n>This paper proposes a dual-stage cross-modal alignment framework to address the limitations of existing methods.<n> Experiments on the Hume-Vidmimic2 dataset demonstrate superior performance with an average Pearson coefficient correlation of 0.51 across six emotion dimensions.
  arXiv  Detail & Related papers  (2025-03-13T17:46:16Z)
• Robust Facial Reactions Generation: An Emotion-Aware Framework with Modality Compensation [27.2792182180834]
  We propose an Emotion-aware Modality Compensatory (EMC) framework. Our framework ensures resilience when faced with missing modality data. It also generates more appropriate emotion-aware reactions via the Emotion-aware Attention (EA) module.
  arXiv  Detail & Related papers  (2024-07-22T17:00:02Z)
• Cross-Attention is Not Enough: Incongruity-Aware Dynamic Hierarchical Fusion for Multimodal Affect Recognition [69.32305810128994]
  Incongruity between modalities poses a challenge for multimodal fusion, especially in affect recognition. We propose the Hierarchical Crossmodal Transformer with Dynamic Modality Gating (HCT-DMG), a lightweight incongruity-aware model. HCT-DMG: 1) outperforms previous multimodal models with a reduced size of approximately 0.8M parameters; 2) recognizes hard samples where incongruity makes affect recognition difficult; 3) mitigates the incongruity at the latent level in crossmodal attention.
  arXiv  Detail & Related papers  (2023-05-23T01:24:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.