A complementary study on PlanGPT: Evaluation with defined Performance Metrics and comparison with a planner

• URL: http://arxiv.org/abs/2606.10489v1
• Date: Tue, 09 Jun 2026 07:07:48 GMT
• Title: A complementary study on PlanGPT: Evaluation with defined Performance Metrics and comparison with a planner
• Abstract summary: A planning problem is defined by a set of objects, an initial state and a desired goal state.<n>We did a complementary study to the state-of-the-art LLM called PlanGPT which was released last year.<n>We discovered that PlanGPT is no better than a Greedy search strategy.
• Score: 0.8491493778771617
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Automated Planning is a subfield of Artificial Intelligence (AI) where the main objective is generating a sequence of actions, known as a plan, that helps us reach a goal state from an initial state. A planning problem is defined by a set of objects, an initial state and a desired goal state. The objective is to compute a plan that'll lead us from the inital state to the goal state. Programs that generate plans are called planners. In this paper, we did a complementary study to the state-of-the-art LLM called PlanGPT which was released last year. We redid some experiments to verify whether planning with LLMs is \textbf{pertinent} and \textbf{worthwhile}. We also check whether the results obtained in the official PlanGPT paper for plan coverage were correct, and we also performed a more comprehensive study on PlanGPT's performance: in our paper PlanGPT's performance was evaluated using two metrics: Plan Cost and Plan Generation Time. The results of planGPT were compared to those produced by a traditional planner for the same plans and same metrics. We discovered that PlanGPT is no better than a Greedy search strategy.

Related papers

• PlanningBench: Generating Scalable and Verifiable Planning Data for Evaluating and Training Large Language Models [52.48858778580074]
  Planning is a fundamental capability for large language models (LLMs)<n>PlanningBench is a framework for generating scalable, diverse, verifiable planning data for both evaluation and training.
  arXiv  Detail & Related papers  (2026-05-20T08:10:15Z)
• Satisficing and Optimal Generalised Planning via Goal Regression (Extended Version) [16.43772461453855]
  Generalised planning (GP) refers to the task of synthesising programs that solve families of related planning problems.<n>We introduce a novel, yet simple method for GP: given a set of training problems, for each problem, compute an optimal plan for each goal atom in some order.<n>We formalise and prove the conditions under which our method is guaranteed to learn valid generalised plans and state space pruning axioms for search.
  arXiv  Detail & Related papers  (2025-11-14T09:16:32Z)
• How Far Are LLMs from Symbolic Planners? An NLP-Based Perspective [2.580765958706854]
  We propose a recovery pipeline consisting of an NLP-based evaluation of the generated plans, along with three stages to recover the plans through NLP manipulation.<n>Our findings reveal no clear evidence of underlying reasoning during plan generation, and that a pipeline comprising an NLP-based analysis of the plans, followed by a recovery mechanism, still falls short of the quality and reliability of classical planners.
  arXiv  Detail & Related papers  (2025-08-02T10:20:52Z)
• Make Planning Research Rigorous Again! [32.54078334699621]
  We argue that rigor should be applied to the current trend of work on planning with large language models.<n>The experience and expertise of the planning community are not just important from a historical perspective.<n>We believe that avoiding such known pitfalls will contribute greatly to the progress in building LLM-based planners.
  arXiv  Detail & Related papers  (2025-05-27T18:51:06Z)
• Hindsight Planner: A Closed-Loop Few-Shot Planner for Embodied Instruction Following [62.10809033451526]
  This work focuses on building a task planner for Embodied Instruction Following (EIF) using Large Language Models (LLMs)<n>We frame the task as a Partially Observable Markov Decision Process (POMDP) and aim to develop a robust planner under a few-shot assumption.<n>Our experiments on the ALFRED dataset indicate that our planner achieves competitive performance under a few-shot assumption.
  arXiv  Detail & Related papers  (2024-12-27T10:05:45Z)
• NATURAL PLAN: Benchmarking LLMs on Natural Language Planning [109.73382347588417]
  We introduce NATURAL PLAN, a realistic planning benchmark in natural language containing 3 key tasks: Trip Planning, Meeting Planning, and Calendar Scheduling. We focus our evaluation on the planning capabilities of LLMs with full information on the task, by providing outputs from tools such as Google Flights, Google Maps, and Google Calendar as contexts to the models.
  arXiv  Detail & Related papers  (2024-06-06T21:27:35Z)
• PDDLEGO: Iterative Planning in Textual Environments [56.12148805913657]
  Planning in textual environments has been shown to be a long-standing challenge even for current models. We propose PDDLEGO that iteratively construct a planning representation that can lead to a partial plan for a given sub-goal. We show that plans produced by few-shot PDDLEGO are 43% more efficient than generating plans end-to-end on the Coin Collector simulation.
  arXiv  Detail & Related papers  (2024-05-30T08:01:20Z)
• Socratic Planner: Self-QA-Based Zero-Shot Planning for Embodied Instruction Following [17.608330952846075]
  Embodied Instruction Following (EIF) is the task of executing natural language instructions by navigating and interacting with objects in interactive environments.<n>A key challenge in EIF is compositional task planning, typically addressed through supervised learning or few-shot in-context learning with labeled data.<n>We introduce the Socratic Planner, a self-QA-based zero-shot planning method that infers an appropriate plan without any further training.
  arXiv  Detail & Related papers  (2024-04-21T08:10:20Z)
• Planning and Acting While the Clock Ticks [15.783791140860528]
  In some problems with time pressure, timing is too tight to complete planning before the first action must be executed. We propose a new problem setting: concurrent planning and execution, in which actions can be dispatched (executed) before planning terminates.
  arXiv  Detail & Related papers  (2024-03-21T19:18:47Z)
• Automating the Generation of Prompts for LLM-based Action Choice in PDDL Planning [59.543858889996024]
  Large language models (LLMs) have revolutionized a large variety of NLP tasks.<n>We show how to leverage an LLM to automatically generate NL prompts from PDDL input.<n>Our NL prompts yield better performance than PDDL prompts and simple template-based NL prompts.
  arXiv  Detail & Related papers  (2023-11-16T11:55:27Z)
• EIPE-text: Evaluation-Guided Iterative Plan Extraction for Long-Form Narrative Text Generation [114.50719922069261]
  We propose a new framework called Evaluation-guided Iterative Plan Extraction for long-form narrative text generation (EIPE-text) EIPE-text has three stages: plan extraction, learning, and inference. We evaluate the effectiveness of EIPE-text in the domains of novels and storytelling.
  arXiv  Detail & Related papers  (2023-10-12T10:21:37Z)
• Robust Hierarchical Planning with Policy Delegation [6.1678491628787455]
  We propose a novel framework and algorithm for hierarchical planning based on the principle of delegation. We show this planning approach is experimentally very competitive to classic planning and reinforcement learning techniques on a variety of domains.
  arXiv  Detail & Related papers  (2020-10-25T04:36:20Z)
• Divide-and-Conquer Monte Carlo Tree Search For Goal-Directed Planning [78.65083326918351]
  We consider alternatives to an implicit sequential planning assumption. We propose Divide-and-Conquer Monte Carlo Tree Search (DC-MCTS) for approximating the optimal plan. We show that this algorithmic flexibility over planning order leads to improved results in navigation tasks in grid-worlds.
  arXiv  Detail & Related papers  (2020-04-23T18:08:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.