Can Vision-Language Models Solve the Shell Game?

• URL: http://arxiv.org/abs/2603.08436v1
• Date: Mon, 09 Mar 2026 14:33:25 GMT
• Title: Can Vision-Language Models Solve the Shell Game?
• Abstract summary: We introduce VET-Bench, a synthetic diagnostic testbed featuring visually identical objects that necessitate tracking throughtemporal continuity.<n>Our method achieves state-of-the-art accuracy exceeding 90% on VET-Bench, demonstrating that Vision-Language Models can reliably solve the video shell-game task end-to-end without external tools.
• Score: 15.221979890090077
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Visual entity tracking is an innate cognitive ability in humans, yet it remains a critical bottleneck for Vision-Language Models (VLMs). This deficit is often obscured in existing video benchmarks by visual shortcuts. We introduce VET-Bench, a synthetic diagnostic testbed featuring visually identical objects that necessitate tracking exclusively through spatiotemporal continuity. Our experiments reveal that current state-of-the-art VLMs perform at or near chance level on VET-Bench, exposing a fundamental limitation: an over-reliance on static frame-level features and a failure to maintain entity representations over time. We provide a theoretical analysis drawing connections to the state-tracking problem, proving that fixed-depth transformer-based VLMs are fundamentally limited in tracking indistinguishable objects without intermediate supervision due to expressivity constraints. To address this, we propose Spatiotemporal Grounded Chain-of-Thought (SGCoT): generating object trajectories as explicit intermediate states. Leveraging Molmo2's object tracking ability, we elicit SGCoT reasoning by fine-tuning on synthesized text-only data for alignment. Our method achieves state-of-the-art accuracy exceeding 90% on VET-Bench, demonstrating that VLMs can reliably solve the video shell-game task end-to-end without external tools. Our code and data are available at https://vetbench.github.io .

Related papers

• Benchmarking Visual State Tracking in Multimodal Video Understanding [67.73383974166462]
  We introduce Visual STAte Tracking benchmark (VSTAT) to diagnose visual state tracking in Multimodal Large Language Models (MLLMs)<n>VSTAT consists of 834 clips drawn from both synthetic and real-world videos, paired with 1,500 questions that cannot be answered from any single frame or short segment.<n>We find that MLLMs reason and track correctly in text, but fail at visually perceiving the events they need to track.
  arXiv  Detail & Related papers  (2026-06-02T17:12:05Z)
• TOC-Bench: A Temporal Object Consistency Benchmark for Video Large Language Models [9.648992690108086]
  Video large language models (Video-LLMs) have made strong progress in general video understanding, but their ability to maintain temporal object consistency remains underexplored.<n>We introduce TOC-Bench, a diagnostic benchmark for evaluating temporal object consistency in Video-LLMs.
  arXiv  Detail & Related papers  (2026-05-11T02:47:59Z)
• Tracking the Truth: Object-Centric Spatio-Temporal Monitoring for Video Large Language Models [154.39583176906893]
  Large language models (MLs) have advanced understanding, they remain highly prone to hallucinations in scenes.<n>We argue this stems from a failure intemporal monitoring, the ability to persistently track object dynamic identities, states, and relations over time.<n>Existing benchmarks obscure this deficit relying on single final-answer queries that can often be resolved via local visual cues or statistical priors.
  arXiv  Detail & Related papers  (2026-05-09T14:32:36Z)
• TransVLM: A Vision-Language Framework and Benchmark for Detecting Any Shot Transitions [52.61846373082384]
  Shot Transition Detection (STD) inherently struggles with complex transitions.<n>We propose TransVLM, a Vision-Language Model (VLM) framework for STD.<n>Extensive experiments demonstrate that TransVLM achieves superior overall performance.
  arXiv  Detail & Related papers  (2026-04-30T15:05:06Z)
• TimeBlind: A Spatio-Temporal Compositionality Benchmark for Video LLMs [24.299498301173255]
  TimeBlind is a diagnostic benchmark for fine-grained temporal understanding.<n>We evaluate over 20 state-of-the-art MLLMs on 600 instances.<n>The Instance Accuracy of the best performing MLLM is only 48.2%, far below the human performance (98.2%)
  arXiv  Detail & Related papers  (2026-01-30T20:21:46Z)
• Assessing Situational and Spatial Awareness of VLMs with Synthetically Generated Video [18.381850705061]
  We introduce a synthetic benchmark that probes two complementary skills: situational awareness and spatial awareness.<n>We test three challenges: distinguishing violence from benign activity, binding assailant roles across viewpoints, and judging fine-grained trajectory alignment.<n>Results show performance only slightly above chance across tasks.
  arXiv  Detail & Related papers  (2026-01-22T09:14:11Z)
• Can LLMs See Without Pixels? Benchmarking Spatial Intelligence from Textual Descriptions [18.455501447828343]
  Spatial Intelligence (SI) has predominantly relied on Vision-Language Models (VLMs)<n>We introduce SiT-Bench, a novel benchmark designed to evaluate the SI performance of Large Language Models (LLMs) without pixel-level input.<n>We find that explicit spatial reasoning significantly boosts performance, suggesting that LLMs possess latent world-modeling potential.
  arXiv  Detail & Related papers  (2026-01-07T05:13:52Z)
• VLNVerse: A Benchmark for Vision-Language Navigation with Versatile, Embodied, Realistic Simulation and Evaluation [61.82502719679122]
  We introduce VLNVerse, a benchmark for Versatile, Embodied, Realistic Simulation, and Evaluation.<n>VLNVerse redefines VLN as a scalable, full-stack embodied AI problem.<n>We propose a novel unified multi-task model capable of addressing all tasks within the benchmark.
  arXiv  Detail & Related papers  (2025-12-22T04:27:26Z)
• TRAVL: A Recipe for Making Video-Language Models Better Judges of Physics Implausibility [70.24211591214528]
  Video generative models produce sequences that violate intuitive physical laws, such as objects floating, teleporting, or morphing.<n>Existing Video-Language Models (VLMs) struggle to identify physics violations, exposing fundamental limitations in their temporal and causal reasoning.<n>We introduce TRAVL, a fine-tuning recipe that combines a balanced training dataset with a trajectory-aware attention module to improve motion encoding.<n>We propose ImplausiBench, a benchmark of 300 videos (150 real, 150 generated) that removes linguistic biases and isolates visual-temporal understanding.
  arXiv  Detail & Related papers  (2025-10-08T21:03:46Z)
• ST-VLM: Kinematic Instruction Tuning for Spatio-Temporal Reasoning in Vision-Language Models [63.12671761097701]
  Vision-Language Models (Ms) struggle to analyze elements like traveled distance and speed of moving objects.<n>We construct a benchmark dataset referred to as STKit and ST-Bench.<n>We show that ST-VLM generalizes robustly across diverse domains and tasks.
  arXiv  Detail & Related papers  (2025-03-25T05:08:06Z)
• TVTSv2: Learning Out-of-the-box Spatiotemporal Visual Representations at Scale [59.01246141215051]
  We analyze the factor that leads to degradation from the perspective of language supervision. We propose a tunable-free pre-training strategy to retain the generalization ability of the text encoder. We produce a series of models, dubbed TVTSv2, with up to one billion parameters.
  arXiv  Detail & Related papers  (2023-05-23T15:44:56Z)
• Learning to Track with Object Permanence [61.36492084090744]
  We introduce an end-to-end trainable approach for joint object detection and tracking. Our model, trained jointly on synthetic and real data, outperforms the state of the art on KITTI, and MOT17 datasets.
  arXiv  Detail & Related papers  (2021-03-26T04:43:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.