CALM-AH: An ABAW11-Calibrated Multimodal Ensemble with Reliability-Gated Multi-Expert Consensus for Video-Level Ambivalence and Hesitancy Recognition

• URL: http://arxiv.org/abs/2607.29310v1
• Date: Fri, 31 Jul 2026 11:37:58 GMT
• Title: CALM-AH: An ABAW11-Calibrated Multimodal Ensemble with Reliability-Gated Multi-Expert Consensus for Video-Level Ambivalence and Hesitancy Recognition
• Abstract summary: The ABAW11 A/H Video Recognition Challenge asks systems to assign a binary A/H label to each naturalistic interview video.<n>We present CALM-AH, a multimodal ensemble that combines textual, acoustic, visual, and derived behavioural-statistical features.<n>We further introduce Reliability-Gated Multi-Expert Consensus(RG-MEC), an anchor-preserving decision-level ensemble that combines an initial prediction with three complementary correction experts.
• Score: 13.270837043204496
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Ambivalence and hesitancy (A/H) are subtle behavioural states that may be expressed through language, voice, facial activity, and other non-verbal cues. The ABAW11 A/H Video Recognition Challenge asks systems to assign a binary A/H label to each naturalistic interview video. Performance is measured using Macro-F1 so that recognition of both A/H and No-A/H samples receives equal importance. We present CALM-AH, a multimodal ensemble that combines textual, acoustic, visual, and derived behavioural-statistical features. We construct 15 non-empty combinations of these feature branches. For each combination, we select the best of three classifier families using validation binary cross-entropy and optimise its decision threshold for validation Macro-F1. The resulting binary decisions are combined using fixed hard-voting weights transferred from BROTHER. We further introduce Reliability-Gated Multi-Expert Consensus(RG-MEC), an anchor-preserving decision-level ensemble that combines an initial prediction with three complementary correction experts: CALM-AH, AffectGPT, and a GPT-based semantic verifier. The initial system provides the default prediction. Its label is overridden only when all three correction experts unanimously support the same alternative class; otherwise, the anchor prediction is retained. This unanimity-gated design limits the influence of isolated expert errors while permitting bidirectional correction when task-specific, multimodal-affective, and semantic-pragmatic evidence are fully consistent. On the participant-disjoint ABAW11 dataset, CALM-AH achieves a Macro-F1 of 0.7525, and the complete RG-MEC system achieves 0.7771.

Related papers

• Knowledge-Guided Multimodal Reasoning over Interacting Streams for Video-Level Ambivalence and Hesitancy Recognition [0.9297355862757838]
  PRISM-AH is a framework that treats A/H as a multimodal conflict that unfolds over time.<n>A knowledge-guided large language model then reasons over structured evidence using the expert cue taxonomy of the dataset.<n>On the labelled public test partition of 525 videos, PRISM-AH attains a macro F1 of 0.6133, compared to the reported zero-shot baseline of 0.2827.
  arXiv  Detail & Related papers  (2026-07-28T16:44:38Z)
• Multimodal Ambivalence and Hesitancy Recognition via Cross-Attention and Gated Fusion [1.6950215926321557]
  We present a framework for Ambivalence/Hesitancy (A/H) recognition in video, developed for the ABAW11 challenge at ECCV 2026.<n>The proposed approach fuses textual, acoustic, and visual modalities extracted from the BAH dataset.<n>This model achieves a Macro F1 of textbf0.7394 on the validation set, a relative improvement of 11.0% over the best unimodal baseline.
  arXiv  Detail & Related papers  (2026-07-17T09:23:36Z)
• Modeling Normal Is All You Need: Joint Latent Clustering for Anomaly Detection in Multimodal Cyber-Physical Systems [47.027290803102666]
  Normal behaviour is the union of many imbalanced, curved, thin-fringed operating regimes.<n>We model the normal law with a jointly learned latent representation plus explicit Gaussian-mixture mode clustering.<n>We evaluate under a deliberately fair protocol: raw point-wise metrics with no point adjustment, a trivial-detector difficulty split, prevalence-matched F1, and train-normal-only calibration.
  arXiv  Detail & Related papers  (2026-07-07T10:10:00Z)
• Dynamic Spectral Denoising with Global-Context Attention for Multi-Behavior Recommendation [58.33732779555114]
  We argue that the key bottleneck is a representation-level failure caused by two coupled heterogeneities.<n>We propose Dynamic Spectral Denoising with Global-Context Attention for Multi-Behavior Recommendation (SpectraMB)<n>SpectraMB achieves the best results in most evaluation settings and exhibits improved robustness under noisy interactions.
  arXiv  Detail & Related papers  (2026-06-01T15:58:13Z)
• AffectAgent: Collaborative Multi-Agent Reasoning for Retrieval-Augmented Multimodal Emotion Recognition [62.16431420189863]
  LLM-based multimodal emotion recognition relies on static parametric memory and often hallucinates when interpreting nuanced affective states.<n>We introduce AffectAgent, an affect-oriented multi-agent retrieval-augmented generation framework.<n>AffectAgent comprises three jointly optimized agents, namely a query planner, an evidence filter, and an emotion generator.
  arXiv  Detail & Related papers  (2026-04-14T13:49:19Z)
• BROTHER: Behavioral Recognition Optimized Through Heterogeneous Ensemble Regularization for Ambivalence and Hesitancy [39.407739937584104]
  Ambivalence and Hesitancy (A/H) in naturalistic video settings remains a significant challenge in affective computing.<n>We propose a highly regularized, multimodal fusion pipeline to predict A/H at the video level.
  arXiv  Detail & Related papers  (2026-03-15T12:53:41Z)
• Feature-level Interaction Explanations in Multimodal Transformers [1.7101146971136896]
  Multimodal Transformers produce predictions without clarifying how different modalities jointly support a decision.<n>We present Feature-level I2MoE, a structured Mixture-of-Experts layer that operates directly on token/patch sequences from frozen pretrained encoders.<n>We develop an expert-wise explanation pipeline that combines attribution with top-K% masking to assess faithfulness.
  arXiv  Detail & Related papers  (2026-03-04T18:24:31Z)
• DiffGRM: Diffusion-based Generative Recommendation Model [63.35379395455103]
  Generative recommendation (GR) is an emerging paradigm that represents each item via a tokenizer as an n-digit semantic ID (SID)<n>We propose DiffGRM, a diffusion-based GR model that replaces the autoregressive decoder with a masked discrete diffusion model (MDM)<n> Experiments show consistent gains over strong generative and discriminative recommendation baselines on multiple datasets.
  arXiv  Detail & Related papers  (2025-10-21T03:23:32Z)
• BiXFormer: A Robust Framework for Maximizing Modality Effectiveness in Multi-Modal Semantic Segmentation [55.486872677160015]
  We reformulate multi-modal semantic segmentation as a mask-level classification task.<n>We propose BiXFormer, which integrates Unified Modality Matching (UMM) and Cross Modality Alignment (CMA)<n> Experiments on both synthetic and real-world multi-modal benchmarks demonstrate the effectiveness of our method.
  arXiv  Detail & Related papers  (2025-06-04T08:04:58Z)
• Trusted Multi-View Classification with Dynamic Evidential Fusion [73.35990456162745]
  We propose a novel multi-view classification algorithm, termed trusted multi-view classification (TMC) TMC provides a new paradigm for multi-view learning by dynamically integrating different views at an evidence level. Both theoretical and experimental results validate the effectiveness of the proposed model in accuracy, robustness and trustworthiness.
  arXiv  Detail & Related papers  (2022-04-25T03:48:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.