Detecting and Mitigating Insertion Hallucination in Video-to-Audio Generation

• URL: http://arxiv.org/abs/2510.08078v3
• Date: Tue, 28 Oct 2025 02:16:25 GMT
• Title: Detecting and Mitigating Insertion Hallucination in Video-to-Audio Generation
• Abstract summary: Video-to-Audio generation has made remarkable strides in automatically sound for video.<n>We term this phenomenon Insertion Hallucination and identify it as a systemic risk driven by dataset biases.<n>We introduce two novel metrics to quantify the prevalence and severity of this issue.<n>We propose Posterior Feature Correction, a novel training-free inference-time method that mitigates IH.
• Score: 29.443084496227026
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Video-to-Audio generation has made remarkable strides in automatically synthesizing sound for video. However, existing evaluation metrics, which focus on semantic and temporal alignment, overlook a critical failure mode: models often generate acoustic events, particularly speech and music, that have no corresponding visual source. We term this phenomenon Insertion Hallucination and identify it as a systemic risk driven by dataset biases, such as the prevalence of off-screen sounds, that remains completely undetected by current metrics. To address this challenge, we first develop a systematic evaluation framework that employs a majority-voting ensemble of multiple audio event detectors. We also introduce two novel metrics to quantify the prevalence and severity of this issue: IH@vid (the fraction of videos with hallucinations) and IH@dur (the fraction of hallucinated duration). Building on this, we propose Posterior Feature Correction, a novel training-free inference-time method that mitigates IH. PFC operates in a two-pass process: it first generates an initial audio output to detect hallucinated segments, and then regenerates the audio after masking the corresponding video features at those timestamps. Experiments on several mainstream V2A benchmarks first reveal that state-of-the-art models suffer from severe IH. In contrast, our PFC method reduces both the prevalence and duration of hallucinations by over 50\% on average, without degrading, and in some cases even improving, conventional metrics for audio quality and temporal synchronization. Our work is the first to formally define, systematically measure, and effectively mitigate Insertion Hallucination, paving the way for more reliable and faithful V2A models.

Related papers

• Video-HolmesV2: Can MLLMs Reason with Spatio-Temporal Audio-Visual Evidence in Long Videos? [22.544360578469156]
  We present Video-Holmes-V2, a novel benchmark for Deep Audio-Visual Coupling.<n>It enforces an Evidence-Based Evaluation, requiring models to justify answers with precise-temporal audio-visual evidence.<n>Our method distills high-value reasoning cues to mitigate long-grained noise.
  arXiv  Detail & Related papers  (2026-09-15T14:30:00Z)
• PRISM-Bench: An Audio-Centric Diagnostic Benchmark for Text-to-Audio-Video Generation [23.127789678422698]
  Text-to-audio-video (T2AV) generation has advanced rapidly, but its evaluation still underestimates the audio modality.<n>We present PRISM-Bench, the first audio-centric diagnostic benchmark for T2AV generation.
  arXiv  Detail & Related papers  (2026-09-04T08:27:10Z)
• CounterFlow: A Two-Phase Inference-Time Sampling for Counterfactual Video Foley Generation [17.978516888210542]
  We present ConterFlow, an inference-time dual-phase sampling scheme for pretrained flow-matching VT2A models.<n>Phase 1 builds a video-derived temporal structure while suppressing the visually implied source.<n>Phase 2 drops video conditioning to focus entirely on shaping audio timbre toward the target prompt.
  arXiv  Detail & Related papers  (2026-05-18T05:42:06Z)
• When Vision Speaks for Sound [31.08404410801052]
  Despite rapid progress in video-capable MLLMs, we find that their apparent audio understanding in videos is often vision-driven.<n>This issue appears across both state-of-the-art open-source omni models and leading closed-source models from providers such as Google and OpenAI.<n>We characterize this failure mode as an audio-visual Clever Hans effect, in which models appear (falsely) audio-grounded, but actually exploit visual-acoustic correlations without verifying whether the audio and visual streams are truly aligned.
  arXiv  Detail & Related papers  (2026-05-13T05:00:19Z)
• Exploring Audio Hallucination in Egocentric Video Understanding [66.1760617001607]
  State-of-the-art large audio-visual language models (AV-LLMs) can generate multimodal descriptions.<n>We show in this work that they are prone to audio hallucinations, often inferring sounds from visual cues that are visible but not heard.
  arXiv  Detail & Related papers  (2026-04-26T20:06:58Z)
• VCE: A zero-cost hallucination mitigation method of LVLMs via visual contrastive editing [70.82867621856968]
  Large vision-language models (LVLMs) frequently suffer from Object Hallucination (OH)<n>Recent studies suggest that the hallucination problem may stem from language priors.<n>We propose Visual Contrastive Editing (VCE), a novel post-hoc method that identifies and suppresses hallucinatory tendencies.
  arXiv  Detail & Related papers  (2026-04-21T12:40:07Z)
• Don't Let the Video Speak: Audio-Contrastive Preference Optimization for Audio-Visual Language Models [57.29797562658234]
  Cross-modal hallucination is bottlenecked by Audio-Visual Language Models (AVLMs)<n>We propose Audio-Contrastive Preference Optimization (ACPO) to counteract this deeply ingrained visual dominance.<n>ACPO introduces an output-contrastive objective to penalize visual descriptions masquerading as audio facts, alongside an input-contrastive objective that swaps audio tracks to explicitly penalize generation invariant to the true auditory signal.
  arXiv  Detail & Related papers  (2026-04-15T17:51:28Z)
• PhyAVBench: A Challenging Audio Physics-Sensitivity Benchmark for Physically Grounded Text-to-Audio-Video Generation [63.3417467957431]
  Text-to-audio-video (T2AV) generation underpins a wide range of applications demanding realistic audio-visual content.<n>We present PhyAVBench, a challenging audio physics-sensitivity benchmark designed to evaluate the audio physics grounding capabilities of existing T2AV models.<n>Unlike prior benchmarks that primarily focus on audio-video synchronization, PhyAVBench explicitly evaluates models' understanding of the physical mechanisms underlying sound generation.
  arXiv  Detail & Related papers  (2025-12-30T05:22:31Z)
• Counting Hallucinations in Diffusion Models [34.45858211220468]
  Diffusion probabilistic models (DPMs) have demonstrated remarkable progress in generative tasks, such as image and video synthesis.<n>They often produce hallucinated samples (hallucinations) that conflict with real-world knowledge.<n>Despite their prevalence, the lack of feasible methodologies for systematically quantifying such hallucinations hinders progress.
  arXiv  Detail & Related papers  (2025-10-15T01:48:04Z)
• Not in Sync: Unveiling Temporal Bias in Audio Chat Models [59.146710538620816]
  Large Audio Language Models (LALMs) are increasingly applied to audio understanding and multimodal reasoning.<n>We present the first systematic study of temporal bias in LALMs, revealing a key limitation in their timestamp prediction.
  arXiv  Detail & Related papers  (2025-10-14T06:29:40Z)
• Mitigating Object Hallucinations via Sentence-Level Early Intervention [10.642552315531404]
  Multimodal large language models (MLLMs) have revolutionized cross-modal understanding but continue to struggle with hallucinations.<n>We propose SENTINEL, a framework that eliminates dependency on human annotations.<n>Sentence-level Early iNtervention Through IN-domain prEference Learning can reduce hallucinations by over 90% compared to the original model.
  arXiv  Detail & Related papers  (2025-07-16T17:55:43Z)
• Generate, but Verify: Reducing Hallucination in Vision-Language Models with Retrospective Resampling [67.14942827452161]
  Vision-Language Models (VLMs) excel at visual understanding but often suffer from visual hallucinations.<n>In this work, we introduce REVERSE, a unified framework that integrates hallucination-aware training with on-the-fly self-verification.
  arXiv  Detail & Related papers  (2025-04-17T17:59:22Z)
• EventHallusion: Diagnosing Event Hallucinations in Video LLMs [42.66453293963568]
  Multimodal Large Language Models (MLLMs) have made significant progress in the video comprehension field.<n>We propose EventHallusion, a novel benchmark that focuses on assessing the VideoLLMs' hallucination toward event.<n>We also propose a simple yet effective method, called Temporal Contrastive Decoding (TCD), to tackle the hallucination problems of VideoLLMs.
  arXiv  Detail & Related papers  (2024-09-25T03:49:46Z)
• VideoHallucer: Evaluating Intrinsic and Extrinsic Hallucinations in Large Video-Language Models [59.05674402770661]
  This work introduces VideoHallucer, the first comprehensive benchmark for hallucination detection in large video-language models (LVLMs) VideoHallucer categorizes hallucinations into two main types: intrinsic and extrinsic, offering further subcategories for detailed analysis.
  arXiv  Detail & Related papers  (2024-06-24T06:21:59Z)
• Anomalous Sound Detection using Audio Representation with Machine ID based Contrastive Learning Pretraining [52.191658157204856]
  This paper uses contrastive learning to refine audio representations for each machine ID, rather than for each audio sample. The proposed two-stage method uses contrastive learning to pretrain the audio representation model. Experiments show that our method outperforms the state-of-the-art methods using contrastive learning or self-supervised classification.
  arXiv  Detail & Related papers  (2023-04-07T11:08:31Z)
• Thinking Hallucination for Video Captioning [0.76146285961466]
  In video captioning, there are two kinds of hallucination: object and action hallucination. We identify three main factors: (i) inadequate visual features extracted from pre-trained models, (ii) improper influences of source and target contexts during multi-modal fusion, and (iii) exposure bias in the training strategy. Our method achieves state-of-the-art performance on the MSR-Video to Text (MSR-VTT) and the Microsoft Research Video Description Corpus (MSVD) datasets.
  arXiv  Detail & Related papers  (2022-09-28T06:15:42Z)
• Audio-visual Representation Learning for Anomaly Events Detection in Crowds [119.72951028190586]
  This paper attempts to exploit multi-modal learning for modeling the audio and visual signals simultaneously. We conduct the experiments on SHADE dataset, a synthetic audio-visual dataset in surveillance scenes. We find introducing audio signals effectively improves the performance of anomaly events detection and outperforms other state-of-the-art methods.
  arXiv  Detail & Related papers  (2021-10-28T02:42:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.