GTR-Bench: Evaluating Geo-Temporal Reasoning in Vision-Language Models

• URL: http://arxiv.org/abs/2510.07791v2
• Date: Fri, 10 Oct 2025 10:28:26 GMT
• Title: GTR-Bench: Evaluating Geo-Temporal Reasoning in Vision-Language Models
• Abstract summary: Geo-Temporal Reasoning benchmark (GTR-Bench) is a novel challenge for geographic temporal reasoning of moving targets in a large-scale camera network.<n> Evaluations of more than 10 popular Visual-Language Models (VLMs) on GTR-Bench demonstrate that even the best proprietary model, Gemini-2.5-Pro, significantly lags behind human performance (78.61%) on geo-temporal reasoning.<n>GTR-Bench offers valuable insights and opens up new opportunities for research and applications in spatial-temporal intelligence.
• Score: 12.634203010453282
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recently spatial-temporal intelligence of Visual-Language Models (VLMs) has attracted much attention due to its importance for Autonomous Driving, Embodied AI and General Artificial Intelligence. Existing spatial-temporal benchmarks mainly focus on egocentric perspective reasoning with images/video context, or geographic perspective reasoning with graphics context (eg. a map), thus fail to assess VLMs' geographic spatial-temporal intelligence with both images/video and graphics context, which is important for areas like traffic management and emergency response. To address the gaps, we introduce Geo-Temporal Reasoning benchmark (GTR-Bench), a novel challenge for geographic temporal reasoning of moving targets in a large-scale camera network. GTR-Bench is more challenging as it requires multiple perspective switches between maps and videos, joint reasoning across multiple videos with non-overlapping fields of view, and inference over spatial-temporal regions that are unobserved by any video context. Evaluations of more than 10 popular VLMs on GTR-Bench demonstrate that even the best proprietary model, Gemini-2.5-Pro (34.9%), significantly lags behind human performance (78.61%) on geo-temporal reasoning. Moreover, our comprehensive analysis on GTR-Bench reveals three primary deficiencies of current models for geo-temporal reasoning. (1) VLMs' reasoning is impaired by an imbalanced utilization of spatial-temporal context. (2) VLMs are weak in temporal forecasting, which leads to worse performance on temporal-emphasized tasks than on spatial-emphasized tasks. (3) VLMs lack the proficiency to comprehend or align the map data with multi-view video inputs. We believe GTR-Bench offers valuable insights and opens up new opportunities for research and applications in spatial-temporal intelligence. Benchmark and code will be released at https://github.com/X-Luffy/GTR-Bench.

Related papers

• Inter-3D VQA: A Roadside Multimodal Benchmark for 3D Spatiotemporally Grounded Visual Question Answering [2.018732483255139]
  We introduce Inter-3D VQA, a large-scale multimodal benchmark for 3D interactions at intersections.<n>Inter-3D VQA contains 407K QA pairs covering lane-level positions, object relationships, motion patterns, and near-miss-oriented interaction reasoning.
  arXiv  Detail & Related papers  (2026-08-28T18:05:58Z)
• GTASA: Ground Truth Annotations for Spatiotemporal Analysis, Evaluation and Training of Video Models [4.964902130083661]
  We introduce GTASA, a corpus of multiactor videos with per-frame spatial- graphs and event-level temporal mappings.<n>We produce it based on Graphs of Events in Space and Time (EST): GEST-Engine.<n>Probing four frozen video encoders across 11temporal reasoning tasks enabled GTASA's exact 3D ground truth.
  arXiv  Detail & Related papers  (2026-04-12T00:01:51Z)
• Spatial-Conditioned Reasoning in Long-Egocentric Videos [17.314383569244303]
  Long-horizon egocentric video presents significant challenges for visual navigation due to viewpoint drift and the absence of persistent geometric context.<n>We study how explicit spatial signals influence VLM-based video understanding without modifying model architectures or inference procedures.
  arXiv  Detail & Related papers  (2026-01-26T03:21:35Z)
• MMSI-Video-Bench: A Holistic Benchmark for Video-Based Spatial Intelligence [61.065486539729875]
  MMSI-Video-Bench is a fully human-annotated benchmark for video-based spatial intelligence in MLLMs.<n>It operationalizes a four-level framework, Perception, Planning, Prediction, and Cross-Video Reasoning, through 1,106 questions grounded in 1,278 clips.<n>We evaluate 25 strong open-source and proprietary MLLMs, revealing a striking human--AI gap.
  arXiv  Detail & Related papers  (2025-12-11T17:57:24Z)
• Multimodal Spatial Reasoning in the Large Model Era: A Survey and Benchmarks [108.15756345836901]
  We provide a comprehensive review of multimodal spatial reasoning tasks with large models.<n>We review advances in embodied AI, including vision-language navigation and action models.<n>We consider emerging modalities such as audio and egocentric video, which contribute to novel spatial understanding through new sensors.
  arXiv  Detail & Related papers  (2025-10-29T17:55:43Z)
• Open-o3 Video: Grounded Video Reasoning with Explicit Spatio-Temporal Evidence [70.2803680525165]
  We introduce Open-o3 Video, a non-agent framework that integrates explicit evidence into video reasoning.<n>The model highlights key objects and bounding boxes alongside its answers, allowing reasoning to be grounded in concrete visual observations.<n>On V-STAR benchmark, Open-o3 Video achieves state-of-the-art performance, raising mAM by 14.4% and mL timestamp by 24.2%.
  arXiv  Detail & Related papers  (2025-10-23T14:05:56Z)
• How Far are VLMs from Visual Spatial Intelligence? A Benchmark-Driven Perspective [103.44502230776352]
  We present a systematic investigation of Visual Spatial Reasoning (VSR) in Vision-Language Models (VLMs)<n>We categorize spatial intelligence into three levels of capability, ie, basic perception, spatial understanding, spatial planning, and curate SIBench, a spatial intelligence benchmark encompassing nearly 20 open-source datasets across 23 task settings.
  arXiv  Detail & Related papers  (2025-09-23T12:00:14Z)
• SIRI-Bench: Challenging VLMs' Spatial Intelligence through Complex Reasoning Tasks [51.774165536666864]
  We introduce SIRI-Bench, a benchmark designed to evaluate Vision-Language Models' structural spatial intelligence.<n>Bench comprises 9,000 video-question-answer triplets, where each problem is embedded in a realistic 3D scene.<n> Experimental results reveal that state-of-the-art VLMs struggle significantly on SIRI-Bench, underscoring the challenge of structural spatial reasoning.
  arXiv  Detail & Related papers  (2025-06-17T13:40:00Z)
• Out of Sight, Not Out of Context? Egocentric Spatial Reasoning in VLMs Across Disjoint Frames [17.975173937253494]
  An embodied AI assistant operating on egocentric video must integrate spatial cues across time.<n>Disjoint-3DQA is a generative QA benchmark that evaluates this ability of VLMs.
  arXiv  Detail & Related papers  (2025-05-30T06:32:26Z)
• Mind the Gap: Benchmarking Spatial Reasoning in Vision-Language Models [14.442394137843923]
  We present a detailed analysis that first delineates the core elements of spatial reasoning.<n>We then assesses the performance of these models in both synthetic and real-world images.
  arXiv  Detail & Related papers  (2025-03-25T14:34:06Z)
• ST-VLM: Kinematic Instruction Tuning for Spatio-Temporal Reasoning in Vision-Language Models [63.12671761097701]
  Vision-Language Models (Ms) struggle to analyze elements like traveled distance and speed of moving objects.<n>We construct a benchmark dataset referred to as STKit and ST-Bench.<n>We show that ST-VLM generalizes robustly across diverse domains and tasks.
  arXiv  Detail & Related papers  (2025-03-25T05:08:06Z)
• Why Is Spatial Reasoning Hard for VLMs? An Attention Mechanism Perspective on Focus Areas [52.478956204238315]
  We study the spatial reasoning challenge from the lens of mechanistic interpretability.<n>We observe that successful spatial reasoning correlates strongly with the model's ability to align its attention with actual object locations.<n>Motivated by these findings, we propose ADAPTVIS to sharpen the attention on highly relevant regions when confident.
  arXiv  Detail & Related papers  (2025-03-03T17:57:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.