ViPlan: A Benchmark for Visual Planning with Symbolic Predicates and Vision-Language Models

• URL: http://arxiv.org/abs/2505.13180v1
• Date: Mon, 19 May 2025 14:38:15 GMT
• Title: ViPlan: A Benchmark for Visual Planning with Symbolic Predicates and Vision-Language Models
• Abstract summary: We introduce ViPlan, the first open-source benchmark for Visual Planning with symbolic predicates and Vision-Language Models (VLMs)<n>We benchmark nine open-source VLM families across multiple sizes, along with selected closed models, evaluating both VLM-grounded symbolic planning and using the models directly to propose actions.<n>We find symbolic planning to outperform direct VLM planning in Blocksworld, where accurate image grounding is crucial, whereas the opposite is true in the household robotics tasks.
• Score: 8.715289208498053
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Integrating Large Language Models with symbolic planners is a promising direction for obtaining verifiable and grounded plans compared to planning in natural language, with recent works extending this idea to visual domains using Vision-Language Models (VLMs). However, rigorous comparison between VLM-grounded symbolic approaches and methods that plan directly with a VLM has been hindered by a lack of common environments, evaluation protocols and model coverage. We introduce ViPlan, the first open-source benchmark for Visual Planning with symbolic predicates and VLMs. ViPlan features a series of increasingly challenging tasks in two domains: a visual variant of the classic Blocksworld planning problem and a simulated household robotics environment. We benchmark nine open-source VLM families across multiple sizes, along with selected closed models, evaluating both VLM-grounded symbolic planning and using the models directly to propose actions. We find symbolic planning to outperform direct VLM planning in Blocksworld, where accurate image grounding is crucial, whereas the opposite is true in the household robotics tasks, where commonsense knowledge and the ability to recover from errors are beneficial. Finally, we show that across most models and methods, there is no significant benefit to using Chain-of-Thought prompting, suggesting that current VLMs still struggle with visual reasoning.

Related papers

• SYMBOLIZER: Symbolic Model-free Task Planning with VLMs [7.564784873669823]
  Traditional Task and Motion Planning (TAMP) systems depend on physics models for motion planning and discrete symbolic models for task planning.<n>Visual Language Models (VLMs) show desirable zero-shot visual understanding.<n>We propose a framework that generalizes well to unseen problem instances.
  arXiv  Detail & Related papers  (2026-04-20T05:32:47Z)
• Using Vision Language Models as Closed-Loop Symbolic Planners for Robotic Applications: A Control-Theoretic Perspective [9.074528675388004]
  Large Language Models (LLMs) and Vision Language Models (VLMs) have been widely used for embodied symbolic planning.<n>We study how to use VLMs as closed-loop symbolic planners for robotic applications from a control-theoretic perspective.
  arXiv  Detail & Related papers  (2025-11-10T18:56:56Z)
• VLMPlanner: Integrating Visual Language Models with Motion Planning [18.633637485218802]
  VLMPlanner is a hybrid framework that combines a learning-based real-time planner with a vision-language model (VLM) capable of reasoning over raw images.<n>We develop the Context-Adaptive Inference Gate mechanism that enables the VLM to mimic human driving behavior.
  arXiv  Detail & Related papers  (2025-07-27T16:15:21Z)
• Gondola: Grounded Vision Language Planning for Generalizable Robotic Manipulation [62.711546725154314]
  We introduce Gondola, a grounded vision-language planning model based on large language models (LLMs) for generalizable robotic manipulation.<n>G Gondola takes multi-view images and history plans to produce the next action plan with interleaved texts and segmentation masks of target objects and locations.<n>G Gondola outperforms the state-of-the-art LLM-based method across all four levels of the GemBench dataset.
  arXiv  Detail & Related papers  (2025-06-12T20:04:31Z)
• Visual Planning: Let's Think Only with Images [30.67065689757505]
  We argue that language may not always be the most natural or effective modality for reasoning, particularly in tasks involving spatial and geometrical information.<n>Motivated by this, we propose a new paradigm, Visual Planning, which enables planning through purely visual representations, independent of text.<n>In this paradigm, planning is executed via sequences of images that encode step-by-step inference in the visual domain, akin to how humans sketch or visualize future actions.
  arXiv  Detail & Related papers  (2025-05-16T16:17:22Z)
• VIPER: Visual Perception and Explainable Reasoning for Sequential Decision-Making [21.61801132083334]
  VIPER is a novel framework for multimodal instruction-based planning.<n>It integrates VLM-based perception with LLM-based reasoning.<n>We show that VIPER significantly outperforms state-of-the-art visual instruction-based planners.
  arXiv  Detail & Related papers  (2025-03-19T11:05:42Z)
• Can-Do! A Dataset and Neuro-Symbolic Grounded Framework for Embodied Planning with Large Multimodal Models [85.55649666025926]
  We introduce Can-Do, a benchmark dataset designed to evaluate embodied planning abilities. Our dataset includes 400 multimodal samples, each consisting of natural language user instructions, visual images depicting the environment, state changes, and corresponding action plans. We propose NeuroGround, a neurosymbolic framework that first grounds the plan generation in the perceived environment states and then leverages symbolic planning engines to augment the model-generated plans.
  arXiv  Detail & Related papers  (2024-09-22T00:30:11Z)
• Wonderful Team: Zero-Shot Physical Task Planning with Visual LLMs [0.0]
  Wonderful Team is a framework for executing high-level robotic planning in a zero-shot regime.<n>We show that Wonderful Team's performance on real-world semantic and physical planning tasks often exceeds methods that rely on separate vision systems.
  arXiv  Detail & Related papers  (2024-07-26T21:18:57Z)
• VSP: Assessing the dual challenges of perception and reasoning in spatial planning tasks for VLMs [102.36953558562436]
  Vision language models (VLMs) are an exciting emerging class of language models (LMs) One understudied capability inVLMs is visual spatial planning. Our study introduces a benchmark that evaluates the spatial planning capability in these models in general.
  arXiv  Detail & Related papers  (2024-07-02T00:24:01Z)
• DKPROMPT: Domain Knowledge Prompting Vision-Language Models for Open-World Planning [9.31108717722043]
  Vision-language models (VLMs) have been applied to robot task planning problems. DKPROMPT automates VLM prompting using domain knowledge in PDDL for classical planning in open worlds.
  arXiv  Detail & Related papers  (2024-06-25T15:49:47Z)
• EgoPlan-Bench: Benchmarking Multimodal Large Language Models for Human-Level Planning [84.6451394629312]
  We introduce EgoPlan-Bench, a benchmark to evaluate the planning abilities of MLLMs in real-world scenarios. We show that EgoPlan-Bench poses significant challenges, highlighting a substantial scope for improvement in MLLMs to achieve human-level task planning. We also present EgoPlan-IT, a specialized instruction-tuning dataset that effectively enhances model performance on EgoPlan-Bench.
  arXiv  Detail & Related papers  (2023-12-11T03:35:58Z)
• Planning as In-Painting: A Diffusion-Based Embodied Task Planning Framework for Environments under Uncertainty [56.30846158280031]
  Task planning for embodied AI has been one of the most challenging problems. We propose a task-agnostic method named 'planning as in-painting' The proposed framework achieves promising performances in various embodied AI tasks.
  arXiv  Detail & Related papers  (2023-12-02T10:07:17Z)
• Compositional Foundation Models for Hierarchical Planning [52.18904315515153]
  We propose a foundation model which leverages expert foundation model trained on language, vision and action data individually together to solve long-horizon tasks. We use a large language model to construct symbolic plans that are grounded in the environment through a large video diffusion model. Generated video plans are then grounded to visual-motor control, through an inverse dynamics model that infers actions from generated videos.
  arXiv  Detail & Related papers  (2023-09-15T17:44:05Z)
• A Picture is Worth a Thousand Words: Language Models Plan from Pixels [53.85753597586226]
  Planning is an important capability of artificial agents that perform long-horizon tasks in real-world environments. In this work, we explore the use of pre-trained language models (PLMs) to reason about plan sequences from text instructions in embodied visual environments.
  arXiv  Detail & Related papers  (2023-03-16T02:02:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.