Team RAS in 10th ABAW Competition: Multimodal Valence and Arousal Estimation Approach

• URL: http://arxiv.org/abs/2603.13056v1
• Date: Fri, 13 Mar 2026 15:06:14 GMT
• Title: Team RAS in 10th ABAW Competition: Multimodal Valence and Arousal Estimation Approach
• Abstract summary: Our method combines three complementary modalities: face, behavior, and audio.<n>Experiments demonstrate that the proposed multimodal fusion strategy achieves a Concordance Correlation Coefficient (CCC) of 0.658 on the Aff-Wild2 development set.
• Score: 53.64487844936037
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Continuous emotion recognition in terms of valence and arousal under in-the-wild (ITW) conditions remains a challenging problem due to large variations in appearance, head pose, illumination, occlusions, and subject-specific patterns of affective expression. We present a multimodal method for valence-arousal estimation ITW. Our method combines three complementary modalities: face, behavior, and audio. The face modality relies on GRADA-based frame-level embeddings and Transformer-based temporal regression. We use Qwen3-VL-4B-Instruct to extract behavior-relevant information from video segments, while Mamba is used to model temporal dynamics across segments. The audio modality relies on WavLM-Large with attention-statistics pooling and includes a cross-modal filtering stage to reduce the influence of unreliable or non-speech segments. To fuse modalities, we explore two fusion strategies: a Directed Cross-Modal Mixture-of-Experts Fusion Strategy that learns interactions between modalities with adaptive weighting, and a Reliability-Aware Audio-Visual Fusion Strategy that combines visual features at the frame-level while using audio as complementary context. The results are reported on the Aff-Wild2 dataset following the 10th Affective Behavior Analysis in-the-Wild (ABAW) challenge protocol. Experiments demonstrate that the proposed multimodal fusion strategy achieves a Concordance Correlation Coefficient (CCC) of 0.658 on the Aff-Wild2 development set.

Related papers

• Team RAS in 11th ABAW Competition: Multimodal Ambivalence Recognition Approach [52.92973028971813]
  We present a single text-centered multimodal approach for video-level ambivalence and hesitancy recognition.<n>The proposed approach combines linguistic, acoustic, facial, and scene features using text-centered multimodal fusion model.<n>Experiments on the Behavioural Ambivalence/Hesitancy (BAH) corpus confirm that text is the strongest unimodal modality.
  arXiv  Detail & Related papers  (2026-07-16T08:04:23Z)
• AdaSurvMamba: Dynamic Fusion and Semantic Scanning for Multimodal Survival Analysis [57.942670844512016]
  We introduce AdaSurvMamba as a novel adaptive framework for multimodal survival analysis.<n>The framework features a Dual-Scale Importance-Aware Reconstruction (DSIR) module to dynamically modulate cross-modal interaction strength.<n>Experiments across five TCGA cohorts demonstrate consistent gains over existing methods.
  arXiv  Detail & Related papers  (2026-06-28T11:36:31Z)
• A Two-Stage Dual-Modality Model for Facial Emotional Expression Recognition [5.014305646574725]
  This paper addresses the expression (EXPR) recognition challenge in the 10th Affective Behavior Analysis in-the-Wild (ABAW) workshop and competition.<n>It requires frame-level classification of eight facial emotional expressions from unconstrained videos.<n>We propose a two-stage dual-modal (audio-visual) model to address these difficulties.
  arXiv  Detail & Related papers  (2026-03-12T17:45:12Z)
• Solution to the 10th ABAW Expression Recognition Challenge: A Robust Multimodal Framework with Safe Cross-Attention and Modality Dropout [20.96260280449496]
  We propose a multimodal framework that dynamically fuses visual and audio representations.<n>Our approach uses a dual-branch Transformer architecture featuring a safe cross-attention mechanism and a modality dropout strategy.<n>This design allows the network to rely on audio-based predictions when visual cues are absent.
  arXiv  Detail & Related papers  (2026-03-09T07:13:20Z)
• Dual-Stream Diffusion for World-Model Augmented Vision-Language-Action Model [62.889356203346985]
  We propose DUal-STream diffusion (DUST), a world-model augmented VLA framework that handles the modality conflict.<n>DUST achieves up to 6% gains over a standard VLA baseline and implicit world-modeling methods.<n>On real-world tasks with the Franka Research 3, DUST outperforms baselines in success rate by 13%.
  arXiv  Detail & Related papers  (2025-10-31T16:32:12Z)
• Interactive Multimodal Fusion with Temporal Modeling [11.506800500772734]
  Our approach integrates visual and audio information through a multimodal framework.<n>The visual branch uses a pre-trained ResNet model to extract features from facial images.<n>The audio branches employ pre-trained VGG models to extract VGGish and LogMel features from speech signals.<n>Our method achieves competitive performance on the Aff-Wild2 dataset, demonstrating effective multimodal fusion for VA estimation in-the-wild.
  arXiv  Detail & Related papers  (2025-03-13T16:31:56Z)
• Dynamic Weighted Combiner for Mixed-Modal Image Retrieval [8.683144453481328]
  Mixed-Modal Image Retrieval (MMIR) as a flexible search paradigm has attracted wide attention. Previous approaches always achieve limited performance, due to two critical factors. We propose a Dynamic Weighted Combiner (DWC) to tackle the above challenges.
  arXiv  Detail & Related papers  (2023-12-11T07:36:45Z)
• Cross-Attention is Not Enough: Incongruity-Aware Dynamic Hierarchical Fusion for Multimodal Affect Recognition [69.32305810128994]
  Incongruity between modalities poses a challenge for multimodal fusion, especially in affect recognition. We propose the Hierarchical Crossmodal Transformer with Dynamic Modality Gating (HCT-DMG), a lightweight incongruity-aware model. HCT-DMG: 1) outperforms previous multimodal models with a reduced size of approximately 0.8M parameters; 2) recognizes hard samples where incongruity makes affect recognition difficult; 3) mitigates the incongruity at the latent level in crossmodal attention.
  arXiv  Detail & Related papers  (2023-05-23T01:24:15Z)
• Modality-Aware Contrastive Instance Learning with Self-Distillation for Weakly-Supervised Audio-Visual Violence Detection [14.779452690026144]
  We propose a modality-aware contrastive instance learning with self-distillation (MACIL-SD) strategy for weakly-supervised audio-visual learning. Our framework outperforms previous methods with lower complexity on the large-scale XD-Violence dataset.
  arXiv  Detail & Related papers  (2022-07-12T12:42:21Z)
• A Joint Cross-Attention Model for Audio-Visual Fusion in Dimensional Emotion Recognition [46.443866373546726]
  We focus on dimensional emotion recognition based on the fusion of facial and vocal modalities extracted from videos. We propose a joint cross-attention model that relies on the complementary relationships to extract the salient features. Our proposed A-V fusion model provides a cost-effective solution that can outperform state-of-the-art approaches.
  arXiv  Detail & Related papers  (2022-03-28T14:09:43Z)
• Dynamic Dual-Attentive Aggregation Learning for Visible-Infrared Person Re-Identification [208.1227090864602]
  Visible-infrared person re-identification (VI-ReID) is a challenging cross-modality pedestrian retrieval problem. Existing VI-ReID methods tend to learn global representations, which have limited discriminability and weak robustness to noisy images. We propose a novel dynamic dual-attentive aggregation (DDAG) learning method by mining both intra-modality part-level and cross-modality graph-level contextual cues for VI-ReID.
  arXiv  Detail & Related papers  (2020-07-18T03:08:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.