VANTAGE-Bench: Evaluating the Infrastructure AI Gap in Vision-Language Models
- URL: http://arxiv.org/abs/2609.09396v1
- Date: Tue, 08 Sep 2026 19:50:25 GMT
- Title: VANTAGE-Bench: Evaluating the Infrastructure AI Gap in Vision-Language Models
- Abstract summary: A class of Physical AI: Infrastructure AI relies on fixed cameras for open-loop logging insights.<n>We introduce VANTAGE-Bench, a benchmark measuring this "Infrastructure AI Gap"<n>We evaluate 17 models zero-shot, we find the shortfall relative to consumer-centric benchmarks is concentrated, not general.
- Score: 6.534399108411788
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: As Vision-Language Models (VLMs) advance toward physical deployment, the focus has remained on action-oriented Embodied AI evaluated on subject-centric consumer video. This overlooks a pervasive class of Physical AI: Infrastructure AI, which relies on fixed cameras for open-loop insights like safety monitoring and operational logging. We introduce VANTAGE-Bench, a benchmark measuring this "Infrastructure AI Gap." It spans three operational domains (Logistics, Transportation, and Smart Spaces), unifies image and video evaluation across semantic, spatial, temporal, and spatio-temporal capabilities, and moves beyond multiple-choice to eight task formulations including dense captioning and spatio-temporal grounding. It adds a single-pass trajectory protocol for Single Object Tracking and, to our knowledge, the first such evaluation on fixed-camera infrastructure video, scored against specialist trackers. Annotation spans three regimes over 3,346 media assets: 3,342 video-task annotations, 4,281 image-grounding annotations, and 27,404 detection boxes. Evaluating 17 models zero-shot, we find the shortfall relative to consumer-centric benchmarks is concentrated, not general. Event verification, referring expressions, and temporal localization fall roughly 9 to 24 points at every model scale, while video question answering stays within 5.3 points of VideoMME and 2D spatial pointing shows no shortfall against BLINK. The temporal pillar is weakest in absolute terms: no system exceeds 55.7 mIoU on temporal localization or 37.3 SODA_c on dense video captioning. On tracking, frontier models come within roughly 5 points of specialist trackers over short horizons but separate as the horizon extends. Open-weight models lead 2D object localization outright, so neither scale nor proprietary access explains the pattern. Data, evaluation harness, and leaderboard: https://vantage-bench.org/
Related papers
- TempoGround: State-Aware Streaming Visual Grounding with Vision-Language Models [7.348671989582267]
Visual grounding maps language referents to targets and is central to openvocabulary perception with vision-language models.<n>We present TempoGround, a VLMnative framework that detects cross-frame object correspondence and explicitly models object presence states.<n>We further introduce Streaming Grounding Reinforcement (SGR), which optimize TempoGround with verifiable Grounding, Identity, and Consistency rewards.
arXiv Detail & Related papers (2026-09-02T09:30:08Z) - TAPVid-MV: A Benchmark for Tracking Any Point in 3D Across Multiple Views [50.26507994908593]
We introduce TAPVid-MV (Tracking Any Point in Video across Multiple Views), the first benchmark for this setting.<n>It contains a curated set of 284 sequences, 1,142 calibrated camera streams, and 109,769 point tracks across seven subsets spanning indoor and outdoor domains.<n>We obtain these trajectories using dataset-specific auxiliary modalities: sensor depth, LiDAR, SLAM and SfM points, human meshes, posed object meshes, and simulation. Every sequence and trajectory is visually verified by human annotators.
arXiv Detail & Related papers (2026-09-01T21:58:51Z) - RSVideo: Are Your Vision-Language Models Ready for Remote Sensing Videos? [24.603802871747092]
We develop a unified evaluation setting for assessing vision-language models on continuous remote-sensing video understanding.<n>We introduce RSVideo, a reinforcement learning framework for small-target focusing that selects question-temporal regions across frames.<n>RSVideo achieves a maximum improvement of 9.01% with InternVL3.5-14B and attains the highest accuracy of 40.63% with Qwen3.6-27B across 26 open-source vision-language backbones.
arXiv Detail & Related papers (2026-08-03T10:34:42Z) - LLaVA-OneVision-2: Towards Next-Generation Perceptual Intelligence [107.63317552620231]
We present LLaVA-OneVision-2 (LLaVA-OV-2), the most capable vision-OneVision model to date.<n>It achieves superior performance across a broad range of multimodal benchmarks.<n>A standout capability is its unified perception across video understanding, temporal grounding, spatial grounding, and manipulation-language reasoning.
arXiv Detail & Related papers (2026-05-25T15:54:04Z) - ARGOS: Who, Where, and When in Agentic Multi-Camera Person Search [49.61286310968402]
We introduce ARGOS, the first benchmark and framework that reformulates multi-camera person search as an interactive reasoning problem.<n>An ARGOS agent receives a vague witness statement and must decide what to ask, when to invoke spatial or temporal tools, and how to interpret ambiguous responses.<n>The benchmark comprises 2,691 tasks across 14 real-world scenarios in three progressive tracks: semantic perception (Who), spatial reasoning (Where), and temporal reasoning (When)
arXiv Detail & Related papers (2026-04-14T14:06:19Z) - MMSI-Video-Bench: A Holistic Benchmark for Video-Based Spatial Intelligence [61.065486539729875]
MMSI-Video-Bench is a fully human-annotated benchmark for video-based spatial intelligence in MLLMs.<n>It operationalizes a four-level framework, Perception, Planning, Prediction, and Cross-Video Reasoning, through 1,106 questions grounded in 1,278 clips.<n>We evaluate 25 strong open-source and proprietary MLLMs, revealing a striking human--AI gap.
arXiv Detail & Related papers (2025-12-11T17:57:24Z) - Open-o3 Video: Grounded Video Reasoning with Explicit Spatio-Temporal Evidence [70.2803680525165]
We introduce Open-o3 Video, a non-agent framework that integrates explicit evidence into video reasoning.<n>The model highlights key objects and bounding boxes alongside its answers, allowing reasoning to be grounded in concrete visual observations.<n>On V-STAR benchmark, Open-o3 Video achieves state-of-the-art performance, raising mAM by 14.4% and mL timestamp by 24.2%.
arXiv Detail & Related papers (2025-10-23T14:05:56Z) - Out of Sight, Not Out of Context? Egocentric Spatial Reasoning in VLMs Across Disjoint Frames [17.975173937253494]
An embodied AI assistant operating on egocentric video must integrate spatial cues across time.<n>Disjoint-3DQA is a generative QA benchmark that evaluates this ability of VLMs.
arXiv Detail & Related papers (2025-05-30T06:32:26Z) - ST-VLM: Kinematic Instruction Tuning for Spatio-Temporal Reasoning in Vision-Language Models [63.12671761097701]
Vision-Language Models (Ms) struggle to analyze elements like traveled distance and speed of moving objects.<n>We construct a benchmark dataset referred to as STKit and ST-Bench.<n>We show that ST-VLM generalizes robustly across diverse domains and tasks.
arXiv Detail & Related papers (2025-03-25T05:08:06Z) - ReferDINO: Referring Video Object Segmentation with Visual Grounding Foundations [33.74746234704817]
Video object segmentation (RVOS) aims to segment target objects throughout a video based on a text description.<n>This is challenging as it involves deep vision-level understanding, pixel-level dense prediction andtemporal reasoning.<n>We propose bfReferDINO RVOS that inherits region-level vision-text alignment from foundational visual grounding models.
arXiv Detail & Related papers (2025-01-24T16:24:15Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.