Multimodal Confidence Modeling in Audio-Visual Quality Assessment

• URL: http://arxiv.org/abs/2605.01219v1
• Date: Sat, 02 May 2026 03:21:05 GMT
• Title: Multimodal Confidence Modeling in Audio-Visual Quality Assessment
• Abstract summary: This paper proposes MCM-AVQA, a multimodal confidence-aware AVQA framework.<n>It estimates modality-specific confidence and injects it into a dedicated audio-visual mixer for cross-modal attention.<n> Experiments show that MCM-AVQA, and specifically its confidence-guided Audio-Visual Mixer, improve correlation with human mean opinion scores.
• Score: 1.7188280334580195
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Audio-visual quality assessment (AVQA) is essential for streaming, teleconferencing, and immersive media. In realistic streaming scenarios, distortions are often asymmetric, where one modality may be severely degraded while the other remains clean. Still, most contemporary AVQA metrics treat audio and video as equally reliable, causing confidence-unaware fusion to emphasize unreliable signals. This paper proposes MCM-AVQA, a multimodal confidence-aware AVQA framework that explicitly estimates modality-specific confidence and injects it into a dedicated audio-visual mixer for cross-modal attention. The Audio-Visual Mixer utilizes frame-level, confidence-guided channel attention to gate fusion, modulating feature interaction between modalities so that high-confidence streams dominate while unreliable inputs are suppressed, preserving temporal degradation patterns. A multi-head visual confidence estimator turns frame-level artifact probabilities into temporally smoothed, clip-level visual confidence scores, while an audio confidence module derives confidence from speech-quality cues without requiring a clean reference. Experiments on multiple AVQA benchmarks show that MCM-AVQA, and specifically its confidence-guided Audio-Visual Mixer, improve correlation with human mean opinion scores and yield more interpretable behavior under real-world asymmetric audio-visual distortions.

Related papers

• Attention-Guided Reliability Scaling for Contrastive Decoding in Robust Audio-Visual Speech Recognition [50.15288513086986]
  Contrastive decoding (CD) originally introduced to stabilize large language model (LLM)-based audio-visual speech recognition (AVSR) systems.<n>We propose reliability-aware scaling of CD for AVSR.
  arXiv  Detail & Related papers  (2026-08-26T08:25:15Z)
• AVBench: Human-Aligned and Automated Evaluation Benchmark for Audio-Video Generative Models [37.78996308837551]
  We introduce AVBench, a fully automated benchmark tailored for human-centric AV generation.<n> AVBench integrates ten evaluation dimensions designed for human-centered real-world scenarios.<n>It derives continuous evaluation scores from the model's prediction confidence on binary decisions.
  arXiv  Detail & Related papers  (2026-05-23T16:42:39Z)
• MSAVBench: Towards Comprehensive and Reliable Evaluation of Multi-Shot Audio-Video Generation [88.7702943548674]
  We introduce MSAVBench, the first comprehensive benchmark and adaptive hybrid evaluation framework for multi-shot audio-video generation.<n>Our benchmark spans four key dimensions, video, audio, shot, and reference, covering diverse task settings, varying shot counts of up to 15, and challenging non-realistic scenarios.<n> MSAVBench achieves high alignment with human judgments, reaching a Spearman rank correlation of 91.5%.
  arXiv  Detail & Related papers  (2026-05-19T17:59:33Z)
• QA-MoE: Towards a Continuous Reliability Spectrum with Quality-Aware Mixture of Experts for Robust Multimodal Sentiment Analysis [9.551550608808752]
  We introduce a Continuous Reliability Spectrum to unify missingness and quality degradation into a single framework.<n>We then propose QA-MoE, a Quality-Aware Mixture-of-Experts framework that quantifies modality reliability via self-supervised aleatoric uncertainty.<n>Extensive experiments indicate that QA-MoE achieves competitive or state-of-the-art performance across diverse degradation scenarios.
  arXiv  Detail & Related papers  (2026-04-07T11:04:23Z)
• Linking Perception, Confidence and Accuracy in MLLMs [15.540652692247098]
  Confidence-Driven Reinforcement Learning (CDRL) uses original-noise image pairs to enhance sensitivity and robustly calibrate the model's confidence.<n> Confidence-Aware Test-Time Scaling (CA-TTS) dynamically coordinates Self-Consistency, Self-Reflection, and Visual Self-Check modules guided by confidence signals.<n>Our integrated framework establishes new state-of-the-art results with consistent 8.8% gains across four benchmarks.
  arXiv  Detail & Related papers  (2026-03-12T16:47:42Z)
• Confidence Estimation for LLMs in Multi-turn Interactions [48.081802290688394]
  This work presents the first systematic study of confidence estimation in multi-turn interactions.<n>We establish a formal evaluation framework grounded in two key desideratas: per-turn calibration and monotonicity of confidence.<n>Our work provides a foundational methodology for developing more reliable and trustworthy conversational agents.
  arXiv  Detail & Related papers  (2026-01-05T14:58:04Z)
• BrowseConf: Confidence-Guided Test-Time Scaling for Web Agents [58.05949210993854]
  We investigate whether search agents have the ability to communicate their own confidence through verbalized confidence scores after long sequences of actions.<n>We propose Test-Time Scaling (TTS) methods that use confidence scores to determine answer quality, encourage the model to try again until reaching a satisfactory confidence level.
  arXiv  Detail & Related papers  (2025-10-27T15:58:51Z)
• Cross-Attention with Confidence Weighting for Multi-Channel Audio Alignment [5.380078543698624]
  Multi-channel audio alignment is a key requirement in bioacoustic monitoring, spatial audio systems, and acoustic localization.<n>We introduce a method that combines cross-attention mechanisms with confidence-weighted scoring to improve multi-channel audio synchronization.<n>Our method achieved first place in the BioDCASE 2025 Task 1 challenge with 0.30 MSE average across test datasets, compared to 0.58 for the deep learning baseline.
  arXiv  Detail & Related papers  (2025-09-21T05:14:06Z)
• AVadCLIP: Audio-Visual Collaboration for Robust Video Anomaly Detection [57.649223695021114]
  We present a novel weakly supervised framework that leverages audio-visual collaboration for robust video anomaly detection.<n>Our framework demonstrates superior performance across multiple benchmarks, with audio integration significantly boosting anomaly detection accuracy.
  arXiv  Detail & Related papers  (2025-04-06T13:59:16Z)
• A Study of Dropout-Induced Modality Bias on Robustness to Missing Video Frames for Audio-Visual Speech Recognition [53.800937914403654]
  Advanced Audio-Visual Speech Recognition (AVSR) systems have been observed to be sensitive to missing video frames. While applying the dropout technique to the video modality enhances robustness to missing frames, it simultaneously results in a performance loss when dealing with complete data input. We propose a novel Multimodal Distribution Approximation with Knowledge Distillation (MDA-KD) framework to reduce over-reliance on the audio modality.
  arXiv  Detail & Related papers  (2024-03-07T06:06:55Z)
• Confidence Estimation for Attention-based Sequence-to-sequence Models for Speech Recognition [31.25931550876392]
  Confidence scores from a speech recogniser are a useful measure to assess the quality of transcriptions. We propose a lightweight and effective approach named confidence estimation module (CEM) on top of an existing end-to-end ASR model.
  arXiv  Detail & Related papers  (2020-10-22T04:02:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.