VISTA: Video Interaction Spatio-Temporal Analysis Benchmark

• URL: http://arxiv.org/abs/2605.01391v1
• Date: Sat, 02 May 2026 11:28:20 GMT
• Title: VISTA: Video Interaction Spatio-Temporal Analysis Benchmark
• Abstract summary: We introduce VISTA, a benchmark for open-set, multi-action-temporal understanding in Vision-Language Models (VLMs)<n>Our benchmark integrates multiple datasets into a single interaction-aware benchmarks and comprises 12K curated video- pairs.<n>Overall, VISTA is the first, large-scale, interaction-aware diagnostic benchmark fortemporal understanding inVLMss.
• Score: 47.42490151556478
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Existing benchmarks for Vision-Language Models (VLMs) primarily evaluate spatio-temporal understanding on simple single-action videos, closed attribute sets and restricted entity types, failing to capture the freeform, multi-action interactions between diverse entities which characterize real-world video understanding. Furthermore, the lack of a systematic framework for analyzing model failures across complementary spatio-temporal axes hinders comprehensive evaluation. To address these gaps, we introduce VISTA, a Video Interaction Spatio-Temporal Analysis benchmark designed for open-set, multi-entity and multi-action spatio-temporal understanding in VLMs. VISTA decomposes videos into interpretable entities, their associated actions, and relational dynamics, enabling multi-axis diagnostics and unified assessment of relational, spatial, and temporal understanding. Our benchmark integrates multiple datasets into a single interaction-aware taxonomy and comprises ~12K curated video-query pairs spanning diverse scenes and complexities. We systematically evaluate 11 state-of-the-art VLMs on VISTA, and break down aggregate performance across our taxonomy to reveal shortcomings and pronounced spatio-temporal biases obscured by traditional metrics. By providing detailed, taxonomy-driven diagnostics on a challenging dataset, VISTA offers a nuanced framework to guide advances in model design, pretraining strategies, and evaluation protocols. Overall, VISTA is the first, large-scale, interaction-aware diagnostic benchmark for spatio-temporal understanding in VLMs.

Related papers

• Inter-X++: A Comprehensive Benchmark for Multimodal Human-Human Interaction Analysis [55.78822230833247]
  We present Inter-X++, a comprehensive and large-scale benchmark designed to empower versatile HHI analysis.<n>Inter-X++ provides 11,388 high-fidelity interaction sequences and over 8.1M frames, featuring precise whole-body movements and detailed finger articulations.<n>We propose OpenHHI, a single and unified HHI representation and modeling framework that jointly optimize interaction reconstruction and semantic understanding.
  arXiv  Detail & Related papers  (2026-08-20T17:51:48Z)
• MultiView-Bench: A Diagnostic Benchmark for World-Centric Multi-View Integration in VLMs [40.317798580235]
  Recent benchmarks for VLMs largely assess single- or limited-view perception, leaving untested the core cognitive ability to integrate observations across viewpoints into a coherent, world-centric (allocentric) 3D mental model.<n>We introduce MultiView-Bench, a diagnostic benchmark expressly designed to evaluate multi-view integration for holistic 3D scene comprehension.<n>We propose ViewNavigator, a multi-agent framework that actively selects informative viewpoints, perceives, and fuses multi-view evidence, improving diverse base models on MultiView-Bench even under a strict budget-matched comparison.
  arXiv  Detail & Related papers  (2026-07-09T22:22:42Z)
• VGenST-Bench: A Benchmark for Spatio-Temporal Reasoning via Active Video Synthesis [23.34626996348395]
  VGenSTBench is a video benchmark that employs generative models to actively synthesize, highly controlled and diverse evaluation scenarios.<n>We establish a comprehensive 3x2x2 video taxonomy, encompassing Spatial Scale Scene, Perspective Dynamics to span diverse scenarios.
  arXiv  Detail & Related papers  (2026-05-21T14:48:35Z)
• Video-MME-v2: Towards the Next Stage in Benchmarks for Comprehensive Video Understanding [98.3098451637867]
  Video-MME-v2 is a comprehensive benchmark designed to rigorously evaluate the robustness and faithfulness of video understanding.<n>To guarantee data quality, Video-MME-v2 is constructed through a rigorously controlled human annotation pipeline.
  arXiv  Detail & Related papers  (2026-04-06T17:59:56Z)
• SOVABench: A Vehicle Surveillance Action Retrieval Benchmark for Multimodal Large Language Models [43.77182813030722]
  We introduce SOVABench, a real-world retrieval benchmark built from surveillance footage centered on vehicle-related actions.<n>SOVABench defines two evaluation protocols (inter-pair and intra-pair) to assess cross-action discrimination and temporal direction understanding.<n>We present a training-free framework for producing interpretable embeddings from MLLM-generated descriptions for both images and videos.
  arXiv  Detail & Related papers  (2026-01-08T10:58:59Z)
• SVAG-Bench: A Large-Scale Benchmark for Multi-Instance Spatio-temporal Video Action Grounding [48.64661382961745]
  We introduce Spatio-temporal Video Action Grounding (SVAG), a novel task that requires models to simultaneously detect, track, and temporally localize all referent objects in videos.<n>To support this task, we construct SVAG-Bench, a large-scale benchmark comprising 688 videos, 19,590 annotated records, and 903 unique verbs.<n> Empirical results show that existing models perform poorly on SVAG, particularly in dense or complex scenes.
  arXiv  Detail & Related papers  (2025-10-14T22:10:49Z)
• HumanVideo-MME: Benchmarking MLLMs for Human-Centric Video Understanding [120.84817886550765]
  Multimodal Large Language Models (MLLMs) have demonstrated significant advances in visual understanding tasks involving both images and videos.<n>Existing human-centric benchmarks predominantly emphasize video generation quality and action recognition, while overlooking essential perceptual and cognitive abilities required in human-centered scenarios.<n>We propose a rigorously curated benchmark designed to provide a more holistic evaluation of MLLMs in human-centric video understanding.
  arXiv  Detail & Related papers  (2025-07-07T11:52:24Z)
• Flattery in Motion: Benchmarking and Analyzing Sycophancy in Video-LLMs [18.07249962240035]
  Video large language models (Video-LLMs) are increasingly integrated into real-world applications that demand grounded multimodal reasoning.<n>Sycophancy, the tendency of these models to align with user input even when it contradicts the visual evidence, undermines their trustworthiness in such contexts.<n>We propose VISE (Video-LLM Sycophancy Benchmarking and Evaluation), the first benchmark designed to evaluate sycophantic behavior in state-of-the-art Video-LLMs.
  arXiv  Detail & Related papers  (2025-06-08T15:00:21Z)
• VideoMolmo: Spatio-Temporal Grounding Meets Pointing [66.19964563104385]
  VideoMolmo is a model tailored for fine-grained pointing of video sequences.<n>A novel temporal mask fusion employs SAM2 for bidirectional point propagation.<n>To evaluate the generalization of VideoMolmo, we introduce VPoMolS-temporal, a challenging out-of-distribution benchmark spanning five real-world scenarios.
  arXiv  Detail & Related papers  (2025-06-05T17:59:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.