Solution for 10th Competition on Ambivalence/Hesitancy (AH) Video Recognition Challenge using Divergence-Based Multimodal Fusion

• URL: http://arxiv.org/abs/2603.16939v1
• Date: Sun, 15 Mar 2026 15:37:25 GMT
• Title: Solution for 10th Competition on Ambivalence/Hesitancy (AH) Video Recognition Challenge using Divergence-Based Multimodal Fusion
• Abstract summary: We address the Ambivalence/Hesitancy (A/H) Video Recognition Challenge at the 10th ABAW Competition (CVPR 2026)<n>We propose a divergence-based multimodal fusion that explicitly measures cross-modal conflict between visual, audio, and textual channels.<n>Our approach achieves a Macro F1 of 0.6808 on the validation test set, outperforming the challenge baseline of 0.2827.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We address the Ambivalence/Hesitancy (A/H) Video Recognition Challenge at the 10th ABAW Competition (CVPR 2026). We propose a divergence-based multimodal fusion that explicitly measures cross-modal conflict between visual, audio, and textual channels. Visual features are encoded as Action Units (AUs) extracted via Py-Feat, audio via Wav2Vec 2.0, and text via BERT. Each modality is processed by a BiLSTM with attention pooling and projected into a shared embedding space. The fusion module computes pairwise absolute differences between modality embeddings, directly capturing the incongruence that characterizes A/H. On the BAH dataset, our approach achieves a Macro F1 of 0.6808 on the validation test set, outperforming the challenge baseline of 0.2827. Statistical analysis across 1{,}132 videos confirms that temporal variability of AUs is the dominant visual discriminator of A/H.

Related papers

• Knowledge-Guided Multimodal Reasoning over Interacting Streams for Video-Level Ambivalence and Hesitancy Recognition [0.9297355862757838]
  PRISM-AH is a framework that treats A/H as a multimodal conflict that unfolds over time.<n>A knowledge-guided large language model then reasons over structured evidence using the expert cue taxonomy of the dataset.<n>On the labelled public test partition of 525 videos, PRISM-AH attains a macro F1 of 0.6133, compared to the reported zero-shot baseline of 0.2827.
  arXiv  Detail & Related papers  (2026-07-28T16:44:38Z)
• Multimodal Ambivalence and Hesitancy Recognition via Cross-Attention and Gated Fusion [1.6950215926321557]
  We present a framework for Ambivalence/Hesitancy (A/H) recognition in video, developed for the ABAW11 challenge at ECCV 2026.<n>The proposed approach fuses textual, acoustic, and visual modalities extracted from the BAH dataset.<n>This model achieves a Macro F1 of textbf0.7394 on the validation set, a relative improvement of 11.0% over the best unimodal baseline.
  arXiv  Detail & Related papers  (2026-07-17T09:23:36Z)
• Team RAS in 11th ABAW Competition: Multimodal Ambivalence Recognition Approach [52.92973028971813]
  We present a single text-centered multimodal approach for video-level ambivalence and hesitancy recognition.<n>The proposed approach combines linguistic, acoustic, facial, and scene features using text-centered multimodal fusion model.<n>Experiments on the Behavioural Ambivalence/Hesitancy (BAH) corpus confirm that text is the strongest unimodal modality.
  arXiv  Detail & Related papers  (2026-07-16T08:04:23Z)
• CF-Net: Conflict Fusion with Speaker Normalisation and Certainty Weighting for Ambivalence/Hesitancy Recognition [4.4644619064521995]
  CF-Net encodes visual, audio, and transcript streams with frozen SigLIP2, HuBERT, and DistilBERT backbones.<n>A ConflictFusion module explicitly computes pairwise cross-modal incongruence.<n> CF-Net achieves a Macro F1 of 0.7155 on the BAH validation set and 0.7364 (AP = 0.7439) on the private challenge test set.
  arXiv  Detail & Related papers  (2026-07-15T16:02:21Z)
• Not All Modalities Are Equal: Instruction-Aware Gating for Multimodal Videos [70.78278988781882]
  We present a unified multimodal video understanding framework, named UniMVU, that performs instruction-aware fusion across video, audio, depth map, or any other modality inputs.<n>Our analysis shows that the gating mechanism aligns with the human-interpretable modality relevance, and ablations show the contributions of inner-modality and modality-level gating.
  arXiv  Detail & Related papers  (2026-05-25T18:02:20Z)
• Separate First, Fuse Later: Mitigating Cross-Modal Interference in Audio-Visual LLMs Reasoning with Modality-Specific Chain-of-Thought [49.53567098922619]
  We propose Separate First, Fuse Later (SFFL) as an audio-visual reasoning framework to reduce cross-modal interference.<n>SFFL enforces modality-specific chain-of-thought reasoning, producing separate audio and visual reasoning traces and integrating evidence for answering.<n> Experiments demonstrate consistent improvements in both accuracy and robustness, yielding an average relative gain of 5.16% on general AVQA benchmarks and 11.17% on a cross-modal hallucination benchmark.
  arXiv  Detail & Related papers  (2026-05-11T02:50:27Z)
• Team RAS in 10th ABAW Competition: Multimodal Valence and Arousal Estimation Approach [53.64487844936037]
  Our method combines three complementary modalities: face, behavior, and audio.<n>Experiments demonstrate that the proposed multimodal fusion strategy achieves a Concordance Correlation Coefficient (CCC) of 0.658 on the Aff-Wild2 development set.
  arXiv  Detail & Related papers  (2026-03-13T15:06:14Z)
• Team LEYA in 10th ABAW Competition: Multimodal Ambivalence/Hesitancy Recognition Approach [43.09390014734485]
  This paper presents a multimodal approach for ambivalence/hesitancy recognition in unconstrained videos.<n>The proposed approach integrates four complementary modalities: scene, face, audio, and text.<n>Experiments on the BAH corpus demonstrate clear gains of multimodal fusion over all unimodal baselines.
  arXiv  Detail & Related papers  (2026-03-13T09:50:03Z)
• Recursive Joint Cross-Modal Attention for Multimodal Fusion in Dimensional Emotion Recognition [3.5803801804085347]
  We introduce Recursive Joint Cross-Modal Attention (RJCMA) to capture both intra- and inter-modal relationships across audio, visual, and text modalities for dimensional emotion recognition. In particular, we compute the attention weights based on cross-correlation between the joint audio-visual-text feature representations and the feature representations of individual modalities. Extensive experiments are conducted to evaluate the performance of the proposed fusion model on the challenging Affwild2 dataset.
  arXiv  Detail & Related papers  (2024-03-20T15:08:43Z)
• MLCA-AVSR: Multi-Layer Cross Attention Fusion based Audio-Visual Speech Recognition [62.89464258519723]
  We propose a multi-layer cross-attention fusion based AVSR approach that promotes representation of each modality by fusing them at different levels of audio/visual encoders. Our proposed approach surpasses the first-place system, establishing a new SOTA cpCER of 29.13% on this dataset.
  arXiv  Detail & Related papers  (2024-01-07T08:59:32Z)
• Cross-Attention is Not Enough: Incongruity-Aware Dynamic Hierarchical Fusion for Multimodal Affect Recognition [69.32305810128994]
  Incongruity between modalities poses a challenge for multimodal fusion, especially in affect recognition. We propose the Hierarchical Crossmodal Transformer with Dynamic Modality Gating (HCT-DMG), a lightweight incongruity-aware model. HCT-DMG: 1) outperforms previous multimodal models with a reduced size of approximately 0.8M parameters; 2) recognizes hard samples where incongruity makes affect recognition difficult; 3) mitigates the incongruity at the latent level in crossmodal attention.
  arXiv  Detail & Related papers  (2023-05-23T01:24:15Z)
• A Joint Cross-Attention Model for Audio-Visual Fusion in Dimensional Emotion Recognition [46.443866373546726]
  We focus on dimensional emotion recognition based on the fusion of facial and vocal modalities extracted from videos. We propose a joint cross-attention model that relies on the complementary relationships to extract the salient features. Our proposed A-V fusion model provides a cost-effective solution that can outperform state-of-the-art approaches.
  arXiv  Detail & Related papers  (2022-03-28T14:09:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.