Video-HolmesV2: Can MLLMs Reason with Spatio-Temporal Audio-Visual Evidence in Long Videos?

• URL: http://arxiv.org/abs/2609.17248v1
• Date: Tue, 15 Sep 2026 14:30:00 GMT
• Title: Video-HolmesV2: Can MLLMs Reason with Spatio-Temporal Audio-Visual Evidence in Long Videos?
• Abstract summary: We present Video-Holmes-V2, a novel benchmark for Deep Audio-Visual Coupling.<n>It enforces an Evidence-Based Evaluation, requiring models to justify answers with precise-temporal audio-visual evidence.<n>Our method distills high-value reasoning cues to mitigate long-grained noise.
• Score: 22.544360578469156
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multimodal Large Language Models have demonstrated impressive video understanding, yet their ability to reason over long-form narratives is often masked by visual-centric evaluations and inefficient context processing. Existing benchmarks over-rely on visual heuristics while marginalizing auditory cues, effectively reducing models to "silent observers" that bypass genuine cross-modal reasoning. Moreover, standard dense sampling creates an evidence-context trade-off: increasing frames to capture evidence inevitably leads to attention distraction and token explosion. To bridge these gaps, we present Video-HolmesV2, a novel benchmark designed for Deep Audio-Visual Coupling. Unlike previous works, it enforces an Evidence-Based Evaluation, requiring models to justify answers with precise spatio-temporal audio-visual evidence, thereby reducing confounding effects of guessing and hallucinated evidence. To support this, we introduce: (1) a Multi-Model Cross-Verification pipeline to ensure task rigor; (2) a Spatio-temporal Evidence-Aware Metric for fine-grained calibration. Furthermore, we propose an Audio-Text Guided Token Compression framework. By fusing task intent with auditory anchors, our method distills high-value reasoning cues to mitigate long-context noise. In our evaluation, even strong proprietary models achieve below 60% accuracy, while our approach outperforms comparable open-source omni-models.

Related papers

• A Closer Look at Failure Modes in Temporal Understanding of Large Audio-Language Models [64.68800440221138]
  Large Audio Language Models (LALMs) achieve strong performance on a variety of audio understanding tasks but struggle with temporal reasoning.<n>We introduce a benchmark with 1,657 questions across three foundational tasks designed for mechanistic analysis.<n>Comparing attention upweighting against scaling, we find that redistributing attention across audio tokens is more effective than increasing audio attention.
  arXiv  Detail & Related papers  (2026-06-16T01:57:56Z)
• When Vision Speaks for Sound [31.08404410801052]
  Despite rapid progress in video-capable MLLMs, we find that their apparent audio understanding in videos is often vision-driven.<n>This issue appears across both state-of-the-art open-source omni models and leading closed-source models from providers such as Google and OpenAI.<n>We characterize this failure mode as an audio-visual Clever Hans effect, in which models appear (falsely) audio-grounded, but actually exploit visual-acoustic correlations without verifying whether the audio and visual streams are truly aligned.
  arXiv  Detail & Related papers  (2026-05-13T05:00:19Z)
• ReasonAudio: A Benchmark for Evaluating Reasoning Beyond Matching in Text-Audio Retrieval [9.400944614656735]
  ReasonAudio is the first reasoning-intensive benchmark for Text-Audio Retrieval.<n>It comprises 1,000 queries and 10,000 composite audio clips across five fundamental reasoning tasks: Negation, Order, Overlap, Duration, and Mix.<n>Our evaluation of ten state-of-the-art models reveals the following findings: All models struggle with reasoning-intensive audio retrieval.
  arXiv  Detail & Related papers  (2026-05-05T04:44:51Z)
• Talker-T2AV: Joint Talking Audio-Video Generation with Autoregressive Diffusion Modeling [59.97930201616015]
  We propose Talker-T2AV, an autoregressive diffusion framework where high-level cross-modal modeling occurs in a shared backbone.<n>A shared autoregressive language model jointly reasons over audio and video in a unified patch-level token space.<n> Experiments on talking portrait benchmarks show Talker-T2AV outperforms dual-branch baselines in lip-sync accuracy, video quality, and audio quality.
  arXiv  Detail & Related papers  (2026-04-26T07:48:47Z)
• Don't Let the Video Speak: Audio-Contrastive Preference Optimization for Audio-Visual Language Models [57.29797562658234]
  Cross-modal hallucination is bottlenecked by Audio-Visual Language Models (AVLMs)<n>We propose Audio-Contrastive Preference Optimization (ACPO) to counteract this deeply ingrained visual dominance.<n>ACPO introduces an output-contrastive objective to penalize visual descriptions masquerading as audio facts, alongside an input-contrastive objective that swaps audio tracks to explicitly penalize generation invariant to the true auditory signal.
  arXiv  Detail & Related papers  (2026-04-15T17:51:28Z)
• VideoZeroBench: Probing the Limits of Video MLLMs with Spatio-Temporal Evidence Verification [73.02304272829785]
  VideoBench is a hierarchical benchmark for long-video answering that rigorously verifies evidence.<n>It comprises 500 manually annotated questions across 13 domains, paired with temporal intervals and spatial bounding boxes as evidence.<n>Experiments show that even GeminiPro-3-Pro correctly answers fewer than 17% of questions under the standard end-to-end QA setting.<n>Results expose a significant gap between surface-level answer correctness and genuine evidence-based reasoning.
  arXiv  Detail & Related papers  (2026-04-02T03:29:43Z)
• Video-R2: Reinforcing Consistent and Grounded Reasoning in Multimodal Language Models [56.851611990473174]
  Reasoning over dynamic visual content remains a central challenge for large language models.<n>We propose a reinforcement learning approach that enhances both temporal precision and reasoning consistency.<n>The resulting model, Video R2, achieves consistently higher TAC, VAS, and accuracy across multiple benchmarks.
  arXiv  Detail & Related papers  (2025-11-28T18:59:58Z)
• Detecting and Mitigating Insertion Hallucination in Video-to-Audio Generation [29.443084496227026]
  Video-to-Audio generation has made remarkable strides in automatically sound for video.<n>We term this phenomenon Insertion Hallucination and identify it as a systemic risk driven by dataset biases.<n>We introduce two novel metrics to quantify the prevalence and severity of this issue.<n>We propose Posterior Feature Correction, a novel training-free inference-time method that mitigates IH.
  arXiv  Detail & Related papers  (2025-10-09T11:08:07Z)
• AURA: A Fine-Grained Benchmark and Decomposed Metric for Audio-Visual Reasoning [3.949628618389608]
  AURA is a benchmark for evaluating the cross-modal reasoning capabilities of Audio-Visual Large Language Models (AV-LLMs) and Omni-modal Language Models (OLMs)<n>AURA includes questions across six challenging cognitive domains, such as causality, timbre and pitch, tempo and AV synchronization, unanswerability, implicit distractions, and skill profiling.<n>We propose a novel metric, AuraScore, which addresses the lack of robust tools for evaluating reasoning fidelity.
  arXiv  Detail & Related papers  (2025-08-10T20:06:42Z)
• Weakly-Supervised Action Detection Guided by Audio Narration [50.4318060593995]
  We propose a model to learn from the narration supervision and utilize multimodal features, including RGB, motion flow, and ambient sound. Our experiments show that noisy audio narration suffices to learn a good action detection model, thus reducing annotation expenses.
  arXiv  Detail & Related papers  (2022-05-12T06:33:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.