Benchmarking Visual State Tracking in Multimodal Video Understanding

• URL: http://arxiv.org/abs/2606.03920v1
• Date: Tue, 02 Jun 2026 17:12:05 GMT
• Title: Benchmarking Visual State Tracking in Multimodal Video Understanding
• Abstract summary: We introduce Visual STAte Tracking benchmark (VSTAT) to diagnose visual state tracking in Multimodal Large Language Models (MLLMs)<n>VSTAT consists of 834 clips drawn from both synthetic and real-world videos, paired with 1,500 questions that cannot be answered from any single frame or short segment.<n>We find that MLLMs reason and track correctly in text, but fail at visually perceiving the events they need to track.
• Score: 67.73383974166462
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Understanding a video requires more than recognizing isolated moments, as humans continuously track entities, states, and events over time. This capacity for visual state tracking is fundamental to video understanding, yet remains underexplored in current evaluations of Multimodal Large Language Models (MLLMs). We introduce Visual STAte Tracking benchmark (VSTAT), a video-based benchmark designed to diagnose visual state tracking in MLLMs. VSTAT consists of 834 clips drawn from both synthetic and real-world videos, paired with 1,500 questions that cannot be answered from any single frame or short segment, requiring continuous perception and integration of events across the entire video stream. Despite their strong performance on existing video benchmarks, we find that state-of-the-art MLLMs perform far below humans and only modestly above answer-prior baselines. To analyze this gap, we compare MLLMs' thinking traces with the underlying video stream to understand why and when MLLMs fail on VSTAT. We find that MLLMs reason and track correctly in text, but fail at visually perceiving the events they need to track. Finally, our preliminary evaluation suggests that recent agentic approaches, including MLLM-based video agents and coding agents, do not readily resolve these failures, still falling short on VSTAT.

Related papers

• Beyond the Single Camera: Agentic Multi-View Reasoning in Sports Video Understanding [21.581174236498196]
  We introduce SportMV-Bench, a benchmark for multi-view sports video understanding.<n>Our analysis shows that current MLLMs fail to effectively exploit multi-view information.<n>We propose SportMV-Agent, an agentic framework that orchestrates an iterative loop of active view selection, perception tool execution, and evidence-grounded reasoning.
  arXiv  Detail & Related papers  (2026-07-13T17:34:36Z)
• GuideMe: Multi-Domain Task Guidance and Intervention in Streaming Video [54.048366682378706]
  We construct GuideMe, the first multi-domain benchmark for streaming video that supports training and evaluation of MLLMs for closed-loop interactive task guidance.<n>It comprises 2,458 videos spanning 223.7 hours across diverse domains (eg, cooking, object manipulation, daily-life guidance, and fitness), with 47,775 interaction samples covering next-step instructions, completion feedback, error detection, and corrective guidance.
  arXiv  Detail & Related papers  (2026-07-03T06:00:04Z)
• Watch, Remember, Reason: Human-View Video Understanding with MLLMs [115.44608894992399]
  Video understanding is being rapidly transformed by large language models (MLLMs)<n>This work presents a human-view on LLM-based video understanding.<n>Rather than treating tasks as isolated benchmarks, this view provides a unified structure for analyzing how MLLMs acquire evidence.
  arXiv  Detail & Related papers  (2026-06-05T16:29:13Z)
• Moment-Video: Diagnosing Temporal Fidelity of Video MLLMs on Momentary Visual Events [52.031070006859544]
  Video multimodal large language models (MLLMs) have made rapid progress on general and long-form video understanding, yet their ability to preserve brief answer-critical visual evidence remains underexplored.<n>We introduce Moment-Video, a benchmark for diagnosing the temporal fidelity of video MLLMs through momentary visual event understanding.
  arXiv  Detail & Related papers  (2026-06-01T17:32:20Z)
• VidVec: Unlocking Video MLLM Embeddings for Video-Text Retrieval [11.519642157641023]
  This paper focuses on leveraging MLLMs for video-text embedding and retrieval.<n>We first conduct a systematic layer-wise analysis, showing that intermediate (pre-trained) MLLM layers already encode substantial task-relevant information.<n>We demonstrate that combining intermediate-layer embeddings with a calibrated MLLM head yields strong zero-shot retrieval performance without any training.
  arXiv  Detail & Related papers  (2026-02-08T19:39:32Z)
• A Survey on Video Temporal Grounding with Multimodal Large Language Model [107.24431595873808]
  Recent advancement in temporal grounding (VTG) has significantly enhanced fine-grained video understanding.<n>With superior multimodal comprehension and reasoning abilities, VTG approaches based on MLLMs (VTG-MLLMs) are gradually surpassing traditional fine-tuned methods.<n>Despite extensive surveys on general video-language understanding, comprehensive reviews specifically addressing VTG-MLLMs remain scarce.
  arXiv  Detail & Related papers  (2025-08-07T08:52:11Z)
• MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding [67.56182262082729]
  We introduce MMBench-Video, a quantitative benchmark to rigorously evaluate large vision-language models (LVLMs) in video understanding. MMBench-Video incorporates lengthy videos from YouTube and employs free-form questions, mirroring practical use cases. The benchmark is meticulously crafted to probe the models' temporal reasoning skills, with all questions human-annotated according to a carefully constructed ability taxonomy.
  arXiv  Detail & Related papers  (2024-06-20T17:26:01Z)
• The Instinctive Bias: Spurious Images lead to Illusion in MLLMs [34.91795817316696]
  We identify a typical class of inputs that baffles MLLMs, which consist of images that are highly relevant but inconsistent with answers. We propose CorrelationQA, the first benchmark that assesses the visual illusion level given spurious images. We conduct a thorough analysis on 9 mainstream MLLMs, illustrating that they universally suffer from this instinctive bias to varying degrees.
  arXiv  Detail & Related papers  (2024-02-06T06:48:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.