Accuracy Without Grounding: Diagnosing Visual Dependency Dissociation in Video LLM Benchmarks

• URL: http://arxiv.org/abs/2607.13305v1
• Date: Tue, 14 Jul 2026 22:15:50 GMT
• Title: Accuracy Without Grounding: Diagnosing Visual Dependency Dissociation in Video LLM Benchmarks
• Abstract summary: We introduce the Visual Dependency Gap (VDG), the difference in per-question correctness between original-video and black-screen conditions.<n>We show that accuracy and visual dependency are separable.<n>These results motivate VDG as a standard audit for whether video benchmarks measure visually grounded capability.
• Score: 2.0789144738685956
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Benchmark accuracy in video large language models (LLMs) is often treated as evidence of visual understanding. We audit this assumption across twenty models spanning 2-78B parameters and ten architecture families. We introduce the Visual Dependency Gap (VDG), the difference in per-question correctness between original-video and black-screen conditions. Paired McNemar tests on MVBench show that accuracy and visual dependency are separable: models differ on original video (p = 0.0003) but not on black screens (p = 0.53). Across models, task-type rankings are stable: Attribute Perception is strongly visual, whereas Temporal Reasoning approaches the language-only baseline. A diagnostic ladder from black screen to single frame, shuffled frames, and original video reveals that frame diversity supplies most of the visual benefit, while temporal order contributes near-zero accuracy across sixteen open-weight models. An ablation from 0.5 to 24 FPS rules out sparse sampling as the cause. H.264 experiments further show that stable aggregate accuracy conceals bidirectional question-level answer flips. The diagnostic also generalizes to four API-accessed models, whose VDG values range from 0.025 to 0.315. These results motivate VDG as a standard audit for whether video benchmarks measure visually grounded capability. Code is available at https://github.com/JaeLee18/accuracy-without-grounding.

Related papers

• The Visual Insensitivity Gap: Diagnosing When Vision-Language Models Fail to Use Visual Evidence [1.7844382164707184]
  We name this phenomenon the Visual Insensitivity Gap and quantify it with a per-sample Visual Sensitivity Index (VSI)<n>We show this assumption fails on 40%--97% of samples across six VLMs and three perceptual benchmarks.
  arXiv  Detail & Related papers  (2026-09-01T08:01:57Z)
• From Accuracy to Visual Dependence: Auditing and Filtering Modality Collapse in Traffic VideoQA [44.46419663487345]
  High benchmark accuracy does not guarantee genuine use of visual evidence.<n>We study this problem in traffic accident Video Question Answering (VideoQA)<n>We propose an instance-level Shortcut Score that combines text-only confidence with visual necessity signals.
  arXiv  Detail & Related papers  (2026-06-29T12:33:15Z)
• Moment-Video: Diagnosing Temporal Fidelity of Video MLLMs on Momentary Visual Events [52.031070006859544]
  Video multimodal large language models (MLLMs) have made rapid progress on general and long-form video understanding, yet their ability to preserve brief answer-critical visual evidence remains underexplored.<n>We introduce Moment-Video, a benchmark for diagnosing the temporal fidelity of video MLLMs through momentary visual event understanding.
  arXiv  Detail & Related papers  (2026-06-01T17:32:20Z)
• COPRA: Conditional Parameter Adaptation with Reinforcement Learning for Video Anomaly Detection [10.332480827185973]
  Vision-language models (VLMs) have shown strong performance in video anomaly detection (VAD)<n>Existing VLM-based VAD methods suffer from a fundamental mismatch between training and inference.<n>We propose COPRA, a conditional parameter adaptation framework for VLM-based VAD.
  arXiv  Detail & Related papers  (2026-05-14T18:39:40Z)
• How Far Can VLMs Go for Visual Bug Detection? Studying 19,738 Keyframes from 41 Hours of Gameplay Videos [41.46982669196867]
  Video-based quality assurance (QA) for long-form gameplay is labor-intensive and error-prone.<n>Recent benchmarks suggest that vision language models (VLMs) can achieve promising results in detecting visual glitches on curated datasets.
  arXiv  Detail & Related papers  (2026-03-24T01:59:06Z)
• TemporalBench: Benchmarking Fine-grained Temporal Understanding for Multimodal Video Models [75.42002690128486]
  TemporalBench is a new benchmark dedicated to evaluating fine-grained temporal understanding in videos. It consists of 10K video question-answer pairs, derived from 2K high-quality human annotations detailing the temporal dynamics in video clips. Results show that state-of-the-art models like GPT-4o achieve only 38.5% question answering accuracy on TemporalBench.
  arXiv  Detail & Related papers  (2024-10-14T17:59:58Z)
• Perception Test: A Diagnostic Benchmark for Multimodal Video Models [78.64546291816117]
  We propose a novel multimodal video benchmark to evaluate the perception and reasoning skills of pre-trained multimodal models. The Perception Test focuses on skills (Memory, Abstraction, Physics, Semantics) and types of reasoning (descriptive, explanatory, predictive, counterfactual) across video, audio, and text modalities. The benchmark probes pre-trained models for their transfer capabilities, in a zero-shot / few-shot or limited finetuning regime.
  arXiv  Detail & Related papers  (2023-05-23T07:54:37Z)
• Making Video Quality Assessment Models Sensitive to Frame Rate Distortions [63.749184706461826]
  We consider the problem of capturing distortions arising from changes in frame rate as part of Video Quality Assessment (VQA) We propose a simple fusion framework, whereby temporal features from GREED are combined with existing VQA models. Our results suggest that employing efficient temporal representations can result much more robust and accurate VQA models.
  arXiv  Detail & Related papers  (2022-05-21T04:13:57Z)
• Deconfounded Video Moment Retrieval with Causal Intervention [80.90604360072831]
  We tackle the task of video moment retrieval (VMR), which aims to localize a specific moment in a video according to a textual query. Existing methods primarily model the matching relationship between query and moment by complex cross-modal interactions. We propose a causality-inspired VMR framework that builds structural causal model to capture the true effect of query and video content on the prediction.
  arXiv  Detail & Related papers  (2021-06-03T01:33:26Z)
• ST-GREED: Space-Time Generalized Entropic Differences for Frame Rate Dependent Video Quality Prediction [63.749184706461826]
  We study how perceptual quality is affected by frame rate, and how frame rate and compression combine to affect perceived quality. We devise an objective VQA model called Space-Time GeneRalized Entropic Difference (GREED) which analyzes the statistics of spatial and temporal band-pass video coefficients. GREED achieves state-of-the-art performance on the LIVE-YT-HFR Database when compared with existing VQA models.
  arXiv  Detail & Related papers  (2020-10-26T16:54:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.