PlanBench-V: A Spatial Planning Map Benchmark for Vision-Language Models

• URL: http://arxiv.org/abs/2606.05744v1
• Date: Thu, 04 Jun 2026 06:17:11 GMT
• Title: PlanBench-V: A Spatial Planning Map Benchmark for Vision-Language Models
• Abstract summary: We introduce PlanBench-V, the first comprehensive benchmark for evaluating Vision-Language Models (VLMs) in spatial planning map interpretation.<n>We first build the Spatial Planning Map Database (SPMD), an expert-annotated dataset of 223 planning maps and 1629 question-answer pairs by professional planners.<n>We then propose a theory-informed evaluation framework assessing four progressive capabilities: Perception, Reasoning, Association, and Implementation.
• Score: 12.535782832272062
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Spatial planning maps are central to territorial governance, translating planning objectives, regulations, and spatial strategies into visual forms for decision-making, public communication, and institutional coordination. Their interpretation, however, requires fine-grained visual perception, spatial reasoning, and policy-informed professional judgment, creating major challenges for both human learners and AI systems. With the rapid progress of Vision-Language Models (VLMs), their use in urban planning analysis is gaining attention, yet existing multimodal benchmarks mainly target general visual understanding and overlook the domain-specific cognitive processes of planning practice. To address this gap, we introduce PlanBench-V, the first comprehensive benchmark for evaluating VLMs in spatial planning map interpretation. We first build the Spatial Planning Map Database (SPMD), an expert-annotated dataset of 223 planning maps and 1629 question-answer pairs curated by professional planners, covering diverse geographic regions and cartographic styles. We then propose a theory-informed evaluation framework assessing four progressive capabilities: Perception, Reasoning, Association, and Implementation, corresponding to the cognitive pipeline of planning map interpretation. Extensive experiments across two generations of VLMs show clear progress but persistent limitations. The best 2026 agentic reasoning model, Qwen3.6-Plus, substantially outperforms the best 2025 model, GPT-4o, by 27%. Nevertheless, all models still struggle with implementation-oriented tasks requiring evaluative judgment, policy sensitivity, and constraint-aware decision-making. These findings reveal fundamental limitations of current VLMs in professional planning contexts and highlight the need for domain-adaptive multimodal reasoning frameworks. Code and data are available at https://plangpt.github.io.

Related papers

• PlanningBench: Generating Scalable and Verifiable Planning Data for Evaluating and Training Large Language Models [52.48858778580074]
  Planning is a fundamental capability for large language models (LLMs)<n>PlanningBench is a framework for generating scalable, diverse, verifiable planning data for both evaluation and training.
  arXiv  Detail & Related papers  (2026-05-20T08:10:15Z)
• Explain Before You Answer: A Survey on Compositional Visual Reasoning [74.27548620675748]
  Compositional visual reasoning has emerged as a key research frontier in multimodal AI.<n>This survey systematically reviews 260+ papers from top venues (CVPR, ICCV, NeurIPS, ICML, ACL, etc.)<n>We then catalog 60+ benchmarks and corresponding metrics that probe compositional visual reasoning along dimensions such as grounding accuracy, chain-of-thought faithfulness, and high-resolution perception.
  arXiv  Detail & Related papers  (2025-08-24T11:01:51Z)
• Scaling Multi-Agent Epistemic Planning through GNN-Derived Heuristics [0.9786469751894747]
  Multi-agent Epistemic Planning (MEP) is an autonomous planning framework for reasoning about both the physical world and the beliefs of agents.<n>The MEP requires states to be represented as Kripke structures, i.e., directed labeled graphs.<n>We exploit Graph Neural Networks (GNNs) to learn patterns and relational structures within epistemic states to guide the planning process.
  arXiv  Detail & Related papers  (2025-08-18T11:26:20Z)
• Wide-Horizon Thinking and Simulation-Based Evaluation for Real-World LLM Planning with Multifaceted Constraints [39.01715254437105]
  This paper introduces the Multiple Aspects of Planning (MAoP) to solve planning problems with multifaceted constraints.<n>Instead of direct planning, MAoP leverages the strategist to conduct pre-planning from various aspects and provide the planning blueprint for planners.
  arXiv  Detail & Related papers  (2025-06-14T09:37:59Z)
• Large Language Models for Planning: A Comprehensive and Systematic Survey [29.168219253281453]
  Large Language Models (LLMs) have demonstrated remarkable performance on certain planning tasks.<n>This paper presents a comprehensive review of LLM-based planning.
  arXiv  Detail & Related papers  (2025-05-26T08:44:53Z)
• PlanGPT-VL: Enhancing Urban Planning with Domain-Specific Vision-Language Models [10.56421857293621]
  We introduce PlanGPT-VL, the first domain-specific Vision-Language Model tailored specifically for urban planning maps.<n>PlanGPT-VL employs three innovative approaches: (1) PlanAnno-V framework for high-quality VQA data synthesis, (2) Critical Point Thinking to reduce hallucinations through structured verification, and (3) comprehensive training methodology combining Supervised Fine-Tuning with frozen vision encoder parameters.
  arXiv  Detail & Related papers  (2025-05-20T15:14:47Z)
• Introduction to AI Planning [0.0]
  The notes begin by introducing the state model and move on to exploring classical planning.<n>The most extensive section is dedicated to Hierarchical Task Network (HTN) planning.<n>The lecture notes end with a bonus chapter on the Planning Domain Definition (PDDL) Language.
  arXiv  Detail & Related papers  (2024-12-16T10:38:04Z)
• On The Planning Abilities of OpenAI's o1 Models: Feasibility, Optimality, and Generalizability [59.72892401927283]
  We evaluate the planning capabilities of OpenAI's o1 models across a variety of benchmark tasks. Our results reveal that o1-preview outperforms GPT-4 in adhering to task constraints.
  arXiv  Detail & Related papers  (2024-09-30T03:58:43Z)
• VSP: Assessing the dual challenges of perception and reasoning in spatial planning tasks for VLMs [102.36953558562436]
  Vision language models (VLMs) are an exciting emerging class of language models (LMs) One understudied capability inVLMs is visual spatial planning. Our study introduces a benchmark that evaluates the spatial planning capability in these models in general.
  arXiv  Detail & Related papers  (2024-07-02T00:24:01Z)
• Planning as In-Painting: A Diffusion-Based Embodied Task Planning Framework for Environments under Uncertainty [56.30846158280031]
  Task planning for embodied AI has been one of the most challenging problems. We propose a task-agnostic method named 'planning as in-painting' The proposed framework achieves promising performances in various embodied AI tasks.
  arXiv  Detail & Related papers  (2023-12-02T10:07:17Z)
• A Planning Ontology to Represent and Exploit Planning Knowledge for Performance Efficiency [6.87593454486392]
  We consider the problem of automated planning, where the objective is to find a sequence of actions that will move an agent from an initial state of the world to a desired goal state. We hypothesize that given a large number of available planners and diverse planning domains; they carry essential information that can be leveraged to identify suitable planners and improve their performance for a domain.
  arXiv  Detail & Related papers  (2023-07-25T14:51:07Z)
• Long-Horizon Visual Planning with Goal-Conditioned Hierarchical Predictors [124.30562402952319]
  The ability to predict and plan into the future is fundamental for agents acting in the world. Current learning approaches for visual prediction and planning fail on long-horizon tasks. We propose a framework for visual prediction and planning that is able to overcome both of these limitations.
  arXiv  Detail & Related papers  (2020-06-23T17:58:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.