Exploring Audio Hallucination in Egocentric Video Understanding

• URL: http://arxiv.org/abs/2604.23860v1
• Date: Sun, 26 Apr 2026 20:06:58 GMT
• Title: Exploring Audio Hallucination in Egocentric Video Understanding
• Abstract summary: State-of-the-art large audio-visual language models (AV-LLMs) can generate multimodal descriptions.<n>We show in this work that they are prone to audio hallucinations, often inferring sounds from visual cues that are visible but not heard.
• Score: 66.1760617001607
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Egocentric videos provide a distinctive setting in which sound serves as crucial cues to understand user activities and surroundings, particularly when visual information is unstable or occluded due to continuous camera movement. State-of-the-art large audio-visual language models (AV-LLMs) can generate multimodal descriptions. However, we show in this work that they are prone to audio hallucinations, often inferring sounds from visual cues that are visible but not heard. We present a systematic and automatic evaluation framework for analyzing audio hallucinations in egocentric video through a targeted question-answering (Q/A) protocol. We curate a dataset of 300 egocentric videos and design 1,000 sound-focused questions to probe model outputs. To characterize hallucinations, we propose a grounded taxonomy that distinguishes between foreground action sounds from the user activities and background ambient sounds. Our evaluation shows that advanced AV-LLMs, such as Qwen2.5 Omni, exhibit high hallucination rates, achieving only 27.3% and 39.5% accuracy on Q/As related to foreground and background sounds, respectively. With this work, we highlight the need to measure the reliability of multimodal responses, emphasizing that robust evaluation of hallucinations is essential to develop reliable AV-LLMs.

Related papers

• When Vision Speaks for Sound [31.08404410801052]
  Despite rapid progress in video-capable MLLMs, we find that their apparent audio understanding in videos is often vision-driven.<n>This issue appears across both state-of-the-art open-source omni models and leading closed-source models from providers such as Google and OpenAI.<n>We characterize this failure mode as an audio-visual Clever Hans effect, in which models appear (falsely) audio-grounded, but actually exploit visual-acoustic correlations without verifying whether the audio and visual streams are truly aligned.
  arXiv  Detail & Related papers  (2026-05-13T05:00:19Z)
• HalluAudio: A Comprehensive Benchmark for Hallucination Detection in Large Audio-Language Models [30.18524844766061]
  We introduce HalluAudio, the first large-scale benchmark for evaluating hallucinations across speech, environmental sound, and music.<n>HalluAudio comprises over 5K human-verified QA pairs and spans diverse task types, including binary judgments, multi-choice reasoning, attribute verification, and open-ended QA.<n>Our results reveal significant deficiencies in acoustic grounding, temporal reasoning, and music attribute understanding, underscoring the need for reliable and robust LALMs.
  arXiv  Detail & Related papers  (2026-04-21T10:05:28Z)
• Don't Let the Video Speak: Audio-Contrastive Preference Optimization for Audio-Visual Language Models [57.29797562658234]
  Cross-modal hallucination is bottlenecked by Audio-Visual Language Models (AVLMs)<n>We propose Audio-Contrastive Preference Optimization (ACPO) to counteract this deeply ingrained visual dominance.<n>ACPO introduces an output-contrastive objective to penalize visual descriptions masquerading as audio facts, alongside an input-contrastive objective that swaps audio tracks to explicitly penalize generation invariant to the true auditory signal.
  arXiv  Detail & Related papers  (2026-04-15T17:51:28Z)
• EgoSound: Benchmarking Sound Understanding in Egocentric Videos [68.1897133235638]
  We introduce EgoSound, the first benchmark designed to evaluate egocentric sound understanding in MLLMs.<n>EgoSound unifies data from Ego4D and EgoBlind, encompassing both sighted and sound-dependent experiences.<n>It defines a seven-task taxonomy spanning intrinsic sound perception, spatial localization, causal inference, and cross-modal reasoning.
  arXiv  Detail & Related papers  (2026-02-15T12:46:35Z)
• JointAVBench: A Benchmark for Joint Audio-Visual Reasoning Evaluation [16.067014259345743]
  We evaluate leading vision-only, audio-only, and Omni-LLMs on our dataset.<n>Even the best-performing Omni-LLM achieves an average accuracy of only 62.6%, outperforming uni-modal baselines.
  arXiv  Detail & Related papers  (2025-12-14T17:23:21Z)
• Can Large Audio-Language Models Truly Hear? Tackling Hallucinations with Multi-Task Assessment and Stepwise Audio Reasoning [55.2480439325792]
  Large audio-language models (LALMs) have shown impressive capabilities in understanding and reasoning about audio and speech information.<n>These models still face challenges, including hallucinating non-existent sound events, misidentifying the order of sound events, and incorrectly attributing sound sources.
  arXiv  Detail & Related papers  (2024-10-21T15:55:27Z)
• VideoHallucer: Evaluating Intrinsic and Extrinsic Hallucinations in Large Video-Language Models [59.05674402770661]
  This work introduces VideoHallucer, the first comprehensive benchmark for hallucination detection in large video-language models (LVLMs) VideoHallucer categorizes hallucinations into two main types: intrinsic and extrinsic, offering further subcategories for detailed analysis.
  arXiv  Detail & Related papers  (2024-06-24T06:21:59Z)
• AdVerb: Visually Guided Audio Dereverberation [49.958724234969445]
  We present AdVerb, a novel audio-visual dereverberation framework. It uses visual cues in addition to the reverberant sound to estimate clean audio.
  arXiv  Detail & Related papers  (2023-08-23T18:20:59Z)
• Egocentric Audio-Visual Noise Suppression [11.113020254726292]
  This paper studies audio-visual noise suppression for egocentric videos. Video camera emulates off-screen speaker's view of the outside world. We first demonstrate that egocentric visual information is helpful for noise suppression.
  arXiv  Detail & Related papers  (2022-11-07T15:53:12Z)
• Thinking Hallucination for Video Captioning [0.76146285961466]
  In video captioning, there are two kinds of hallucination: object and action hallucination. We identify three main factors: (i) inadequate visual features extracted from pre-trained models, (ii) improper influences of source and target contexts during multi-modal fusion, and (iii) exposure bias in the training strategy. Our method achieves state-of-the-art performance on the MSR-Video to Text (MSR-VTT) and the Microsoft Research Video Description Corpus (MSVD) datasets.
  arXiv  Detail & Related papers  (2022-09-28T06:15:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.