When Vision Speaks for Sound

• URL: http://arxiv.org/abs/2605.16403v1
• Date: Wed, 13 May 2026 05:00:19 GMT
• Title: When Vision Speaks for Sound
• Abstract summary: Despite rapid progress in video-capable MLLMs, we find that their apparent audio understanding in videos is often vision-driven.<n>This issue appears across both state-of-the-art open-source omni models and leading closed-source models from providers such as Google and OpenAI.<n>We characterize this failure mode as an audio-visual Clever Hans effect, in which models appear (falsely) audio-grounded, but actually exploit visual-acoustic correlations without verifying whether the audio and visual streams are truly aligned.
• Score: 31.08404410801052
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Despite rapid progress in video-capable MLLMs, we find that their apparent audio understanding in videos is often vision-driven: models rely on visual cues to infer or hallucinate acoustic information, rather than verifying the audio stream. This issue appears across both state-of-the-art open-source omni models and leading closed-source models from providers such as Google and OpenAI. We characterize this failure mode as an audio-visual Clever Hans effect, in which models appear (falsely) audio-grounded, but actually exploit visual-acoustic correlations without verifying whether the audio and visual streams are truly aligned. To systematically study this behavior, we introduce Thud, an intervention-driven probing framework based on three counterfactual audio edits: Shift, which tests temporal synchronization; Mute, which tests sound existence; and Swap, which tests audio-visual consistency. Beyond diagnosis, we further study a two-stage alignment recipe: intervention-derived preference pairs teach audio verification, while event-level general video preferences regularize the model against over-specialization. Our best 10K-sample recipe improves average performance across the three intervention dimensions by 28 percentage points, while slightly improving performance on general video and audio-visual QA benchmarks.

Related papers

• Exploring Audio Hallucination in Egocentric Video Understanding [66.1760617001607]
  State-of-the-art large audio-visual language models (AV-LLMs) can generate multimodal descriptions.<n>We show in this work that they are prone to audio hallucinations, often inferring sounds from visual cues that are visible but not heard.
  arXiv  Detail & Related papers  (2026-04-26T20:06:58Z)
• AudioCapBench: Quick Evaluation on Audio Captioning across Sound, Music, and Speech [56.08149157180447]
  We introduce AudioCapBench, a benchmark for evaluating audio captioning capabilities of large multimodal models.<n>We evaluate 13 models across two providers (OpenAI, Google Gemini) using both reference-based metrics (METEOR, BLEU, ROUGE-L) and an LLM-as-Judge framework.
  arXiv  Detail & Related papers  (2026-02-27T03:33:37Z)
• In-Context Audio Control of Video Diffusion Transformers [28.911323185865186]
  This paper introduces In-Context Audio Control of video diffusion transformers (ICAC)<n>We investigate the integration of audio signals for speech-driven video generation within a unified full-attention architecture, akin to FullDiT.<n>We propose a Masked 3D Attention mechanism that constrains the attention pattern to enforce temporal alignment, enabling stable training and superior performance.
  arXiv  Detail & Related papers  (2025-12-21T15:22:28Z)
• JointAVBench: A Benchmark for Joint Audio-Visual Reasoning Evaluation [16.067014259345743]
  We evaluate leading vision-only, audio-only, and Omni-LLMs on our dataset.<n>Even the best-performing Omni-LLM achieves an average accuracy of only 62.6%, outperforming uni-modal baselines.
  arXiv  Detail & Related papers  (2025-12-14T17:23:21Z)
• Detecting and Mitigating Insertion Hallucination in Video-to-Audio Generation [29.443084496227026]
  Video-to-Audio generation has made remarkable strides in automatically sound for video.<n>We term this phenomenon Insertion Hallucination and identify it as a systemic risk driven by dataset biases.<n>We introduce two novel metrics to quantify the prevalence and severity of this issue.<n>We propose Posterior Feature Correction, a novel training-free inference-time method that mitigates IH.
  arXiv  Detail & Related papers  (2025-10-09T11:08:07Z)
• Exploring Audio Cues for Enhanced Test-Time Video Model Adaptation [46.29811604867483]
  Test-time adaptation (TTA) aims to boost the generalization capability of a trained model by conducting self-/unsupervised learning during the testing phase.<n>We propose a novel approach that incorporates audio information into video TTA.<n>Our method capitalizes on the rich semantic content of audio to generate audio-assisted pseudo-labels.
  arXiv  Detail & Related papers  (2025-06-14T12:44:58Z)
• AdVerb: Visually Guided Audio Dereverberation [49.958724234969445]
  We present AdVerb, a novel audio-visual dereverberation framework. It uses visual cues in addition to the reverberant sound to estimate clean audio.
  arXiv  Detail & Related papers  (2023-08-23T18:20:59Z)
• AV-data2vec: Self-supervised Learning of Audio-Visual Speech Representations with Contextualized Target Representations [88.30635799280923]
  We introduce AV-data2vec which builds audio-visual representations based on predicting contextualized representations. Results on LRS3 show that AV-data2vec consistently outperforms existing methods with the same amount of data and model size.
  arXiv  Detail & Related papers  (2023-02-10T02:55:52Z)
• MAViL: Masked Audio-Video Learners [68.61844803682145]
  We present Masked Audio-Video learners (MAViL) to train audio-visual representations. Pre-training with MAViL enables the model to perform well in audio-visual classification and retrieval tasks. For the first time, a self-supervised audio-visual model outperforms ones that use external supervision on benchmarks.
  arXiv  Detail & Related papers  (2022-12-15T18:59:59Z)
• AudioVisual Video Summarization [103.47766795086206]
  In video summarization, existing approaches just exploit the visual information while neglecting the audio information. We propose to jointly exploit the audio and visual information for the video summarization task, and develop an AudioVisual Recurrent Network (AVRN) to achieve this.
  arXiv  Detail & Related papers  (2021-05-17T08:36:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.