OmniMapBench: Benchmarking Visual-Centric Reasoning on Diverse Map Documents

• URL: http://arxiv.org/abs/2607.09068v1
• Date: Fri, 10 Jul 2026 03:27:25 GMT
• Title: OmniMapBench: Benchmarking Visual-Centric Reasoning on Diverse Map Documents
• Abstract summary: Recent advancements in LVLMs necessitate robust benchmarks for complex, visually grounded reasoning.<n>To address this limitation, OmniMapBench is introduced to foster visual-centric reasoning for map documents.<n>The benchmark comprises 2,096 manually annotated question-answer pairs across 1,603 map documents from nine categories.
• Score: 34.58999637034811
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent advancements in LVLMs necessitate robust benchmarks for complex, visually grounded reasoning. A critical limitation is identified in many document understanding benchmarks: visual content is often reducible to text, enabling high performance without genuine visual grounding. To address this limitation, OmniMapBench is introduced to foster visual-centric reasoning for map documents. The benchmark comprises 2,096 manually annotated question-answer pairs across 1,603 map documents from nine categories. It is designed to probe a hierarchy of skills, ranging from perception to multi-step visual reasoning. To quantify benchmark properties, a simple yet effective benchmark-level metric is proposed: the Visual Dependency Index (VDI), defined as the accuracy drop when images are replaced with question-agnostic descriptions. OmniMapBench exhibits higher VDI than established benchmarks, which quantitatively validates its focus on irreducible visual reasoning. Comprehensive evaluations of 25 leading LVLMs are conducted on OmniMapBench. A significant performance gap is observed, with the top-performing model achieving only 75.03\% accuracy. This result underscores the challenges posed by OmniMapBench to current LVLMs. This work aims to catalyze progress in visual-centric reasoning for document understanding of LVLMs. The dataset and code are publicly available at https://github.com/SIGMME/OmniMapBench.

Related papers

• GamiBench: Evaluating Spatial Reasoning and 2D-to-3D Planning Capabilities of MLLMs with Origami Folding Tasks [2.037026133182214]
  We introduce GamiBench, a benchmark to evaluate spatial reasoning in large language models (MLLMs) through origami-inspired folding tasks.<n>GamiBench includes 186 regular and impossible 2D crease patterns paired with their corresponding 3D folded shapes.<n>GamiBench holistically evaluates the entire reasoning process--measuring cross-view consistency, physical feasibility through impossible-fold detection, and interpretation of intermediate folding steps.
  arXiv  Detail & Related papers  (2025-12-22T01:07:59Z)
• Visually Prompted Benchmarks Are Surprisingly Fragile [82.98001690512461]
  Key challenge in evaluating VLMs is testing their ability to analyze visual content independently from their textual priors.<n>We demonstrate how details in benchmark setup, including visual marker design and dataset size, have a significant influence on model performance and leaderboard rankings.<n>To mitigate this instability, we curate existing datasets to create VPBench, a larger visually prompted benchmark with 16 visual marker variants.
  arXiv  Detail & Related papers  (2025-12-19T18:26:58Z)
• CartoMapQA: A Fundamental Benchmark Dataset Evaluating Vision-Language Models on Cartographic Map Understanding [5.925837407110905]
  We introduce CartoMapQA, a benchmark to evaluate Visual-Language Models' understanding of cartographic maps.<n>The dataset includes over 2000 samples, each composed of a cartographic map, a question (with open-ended or multiple-choice answers), and a ground-truth answer.
  arXiv  Detail & Related papers  (2025-12-03T08:25:22Z)
• Traceable Evidence Enhanced Visual Grounded Reasoning: Evaluation and Methodology [87.65242416688146]
  TreeBench is a diagnostic benchmark for visual grounded reasoning.<n>TreeVGR is a training paradigm to supervise localization and reasoning jointly with reinforcement learning.
  arXiv  Detail & Related papers  (2025-07-10T17:59:58Z)
• Can MLLMs Guide Me Home? A Benchmark Study on Fine-Grained Visual Reasoning from Transit Maps [56.76175383189738]
  We introduce ReasonMap, a benchmark designed to assess the fine-grained visual understanding and spatial reasoning abilities of MLLMs.<n>ReasonMap encompasses high-resolution transit maps from 30 cities across 13 countries and includes 1,008 question-answer pairs spanning two question types and three templates.<n> Comprehensive evaluations of 15 popular MLLMs, including both base and reasoning variants, reveal a counterintuitive pattern.
  arXiv  Detail & Related papers  (2025-05-24T12:33:52Z)
• Are Multimodal Large Language Models Ready for Omnidirectional Spatial Reasoning? [66.88619941063048]
  We ask: Are multimodal large language models (MLLMs) ready for omnidirectional spatial reasoning?<n> OSR-Bench is the first benchmark specifically designed for this setting.<n>It includes over 153,000 diverse question-answer pairs grounded in high-fidelity panoramic indoor scene maps.<n>We evaluate eight state-of-the-art MLLMs, including GPT-4o, Gemini 1.5 Pro, and leading open-source models under zero-shot settings.
  arXiv  Detail & Related papers  (2025-05-17T08:48:40Z)
• Can Large Vision Language Models Read Maps Like a Human? [16.81757312518894]
  MapBench comprises over 1600 pixel space map path finding problems from 100 diverse maps.<n>In MapBench, LVLMs generate language-based navigation instructions given a map image and a query with beginning and end landmarks.<n>We demonstrate that MapBench significantly challenges state-of-the-art LVLMs both zero-shot prompting and a Chain-of-Thought (CoT) augmented reasoning framework.
  arXiv  Detail & Related papers  (2025-03-18T18:05:38Z)
• ZeroBench: An Impossible Visual Benchmark for Contemporary Large Multimodal Models [79.90955995669158]
  Large Multimodal Models (LMMs) exhibit major shortfalls when interpreting images and, by some measures, have poorer spatial cognition than small children or animals.<n>We introduce ZeroBench, a lightweight visual reasoning benchmark that is entirely impossible for contemporary frontier LMMs.<n>We evaluate 20 LMMs on ZeroBench, all of which score 0.0%, and rigorously analyse the errors.
  arXiv  Detail & Related papers  (2025-02-13T18:59:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.