Audio-Text Cross-Attention with Psycholinguistic Support Features for Ambivalence/Hesitancy Recognition

• URL: http://arxiv.org/abs/2607.13345v1
• Date: Wed, 15 Jul 2026 00:11:24 GMT
• Title: Audio-Text Cross-Attention with Psycholinguistic Support Features for Ambivalence/Hesitancy Recognition
• Abstract summary: We present an audio-text system for the Ambivalence/Hesitancy Video Recognition Challenge of the 11th ABAW Competition.<n>The method excludes visual frames and represents each video as overlapping 5-second aligned with transcript timestamps.<n>Each window combines a 320-dimensional prosodic audio descriptor, a 768-dimensional emotion-oriented RoBERTa embedding, and 74 handcrafted features capturing uncertainty, hedging, and attitudinal conflict.
• Score: 0.6042845803090501
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present an audio-text system for the Ambivalence/Hesitancy Video Recognition Challenge of the 11th ABAW Competition. The method excludes visual frames and represents each video as overlapping 5-second windows aligned with transcript timestamps. Each window combines a 320-dimensional prosodic audio descriptor, a 768-dimensional emotion-oriented RoBERTa embedding, and 74 handcrafted features capturing uncertainty, hedging, and attitudinal conflict. Audio and text are fused via temporal cross-attention, while support features are injected prior to gated multiple-instance learning (MIL) pooling to modulate the window's importance. Predictions from five independently initialized models are averaged. On the labeled public development set, the ensemble achieved an average precision of 0.875 and a macro-F1 of 0.72. Our source code is publicly available at https://github.com/Liga-de-IA-PUCPR/abaw-11-ah-challenge/.

Related papers

• Long-Horizon Audio-Visual Generation for Persistent Stories and Interactive Worlds [60.18437892555415]
  We present JoyAI-Echo-1.5, a unified audio-visual generation system with two purpose-built variants.<n>The long-video variant introduces composable cross-shot memory that aggregates visual evidence across multiple prior shots and speaker cues derived from speech-filtered full-shot audio.<n>The world-model variant converts heterogeneous navigation inputs into calibrated metric 6-DoF camera trajectories and injects them through a geometry-aware conditioning pathway.
  arXiv  Detail & Related papers  (2026-08-24T15:31:12Z)
• OmniVideo-100K: A Dataset for Audio-Visual Reasoning through Structured Scripts and Evidence Chains [50.186434778589415]
  Current automated pipelines for audio-visual Question Answering (QA) generally adopt a video-caption-QA'' paradigm.<n>We propose an automated data engine featuring two mechanisms: textbfEntity-Anchored Video Scripting' and textbfClue-Guided QA Generation'
  arXiv  Detail & Related papers  (2026-06-12T17:59:55Z)
• Speed by Simplicity: A Single-Stream Architecture for Fast Audio-Video Generative Foundation Model [52.79436545460808]
  We present daVinci-MagiHuman, an open-source audio-video generative foundation model for human-centric generation.<n>DaVinci-MagiHuman jointly generates synchronized video and audio using a single-stream Transformer.<n>It supports multilingual spoken generation across Chinese (Mandarin and Cantonese), English, Japanese, Korean, German, and French.
  arXiv  Detail & Related papers  (2026-03-23T13:49:06Z)
• LVOmniBench: Pioneering Long Audio-Video Understanding Evaluation for Omnimodal LLMs [68.35684758116453]
  This dataset comprises high-quality videos sourced from open platforms that feature rich audio-visual dynamics.<n>We rigorously evaluate the capabilities of OmniLLMs across domains, including long-term memory, temporal localization, fine-grained understanding, and multimodal perception.<n>Open-source models generally achieve accuracies below 35%, whereas the Gemini 3 Pro reaches a peak accuracy of approximately 65%.
  arXiv  Detail & Related papers  (2026-03-19T17:58:13Z)
• Solution for 10th Competition on Ambivalence/Hesitancy (AH) Video Recognition Challenge using Divergence-Based Multimodal Fusion [0.0]
  We address the Ambivalence/Hesitancy (A/H) Video Recognition Challenge at the 10th ABAW Competition (CVPR 2026)<n>We propose a divergence-based multimodal fusion that explicitly measures cross-modal conflict between visual, audio, and textual channels.<n>Our approach achieves a Macro F1 of 0.6808 on the validation test set, outperforming the challenge baseline of 0.2827.
  arXiv  Detail & Related papers  (2026-03-15T15:37:25Z)
• CourseTimeQA: A Lecture-Video Benchmark and a Latency-Constrained Cross-Modal Fusion Method for Timestamped QA [3.0867286767516937]
  Given a natural-language query, the system retrieves relevant timestamped segments and synthesizes a grounded answer.<n>We present CourseTimeQA (52.3 h, 902 queries across six courses) and a lightweight, latency-constrained cross-modal retriever (CrossFusion-RAG)<n>On CourseTimeQA, CrossFusion-RAG improves nDCG@10 by 0.10 and MRR by 0.08 over a strong BLIP-2 retriever while achieving approximately 1.55 s median end-to-end latency on a single A100.
  arXiv  Detail & Related papers  (2025-11-29T07:06:51Z)
• Exploiting Temporal Audio-Visual Correlation Embedding for Audio-Driven One-Shot Talking Head Animation [62.218932509432314]
  Inherently, the temporal relationship of adjacent audio clips is highly correlated with that of the corresponding adjacent video frames.<n>We learn audio-visual correlations and integrate the correlations to help enhance feature representation and regularize final generation.
  arXiv  Detail & Related papers  (2025-04-08T07:23:28Z)
• DeepSound-V1: Start to Think Step-by-Step in the Audio Generation from Videos [4.452513686760606]
  We propose a framework for audio generation from videos, leveraging the internal chain-of-thought (CoT) of a multi-modal large language model (MLLM)<n>A corresponding multi-modal reasoning dataset is constructed to facilitate the learning of initial reasoning in audio generation.<n>In experiments, we demonstrate the effectiveness of the proposed framework in reducing misalignment (voice-over) in generated audio.
  arXiv  Detail & Related papers  (2025-03-28T07:56:19Z)
• Progressive Confident Masking Attention Network for Audio-Visual Segmentation [7.864898315909104]
  A challenging problem known as Audio-Visual (AVS) has emerged, intending to produce segmentation maps for sounding objects within a scene.<n>We introduce a novel Progressive Confident Masking Attention Network (PMCANet)<n>It leverages attention mechanisms to uncover the intrinsic correlations between audio signals and visual frames.
  arXiv  Detail & Related papers  (2024-06-04T14:21:41Z)
• Mirasol3B: A Multimodal Autoregressive model for time-aligned and contextual modalities [67.89368528234394]
  One of the main challenges of multimodal learning is the need to combine heterogeneous modalities. Video and audio are obtained at much higher rates than text and are roughly aligned in time. Our approach achieves the state-of-the-art on well established multimodal benchmarks, outperforming much larger models.
  arXiv  Detail & Related papers  (2023-11-09T19:15:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.