Can LLM-Reasoning Models Replace Classical Planning? A Benchmark Study

• URL: http://arxiv.org/abs/2507.23589v1
• Date: Thu, 31 Jul 2025 14:25:54 GMT
• Title: Can LLM-Reasoning Models Replace Classical Planning? A Benchmark Study
• Authors: Kai Goebel, Patrik Zips,
• Abstract summary: Large Language Models have sparked interest in their potential for robotic task planning.<n>While these models demonstrate strong generative capabilities, their effectiveness in producing structured and executable plans remains uncertain.<n>This paper presents a systematic evaluation of a broad spectrum of current state of the art language models.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent advancements in Large Language Models have sparked interest in their potential for robotic task planning. While these models demonstrate strong generative capabilities, their effectiveness in producing structured and executable plans remains uncertain. This paper presents a systematic evaluation of a broad spectrum of current state of the art language models, each directly prompted using Planning Domain Definition Language domain and problem files, and compares their planning performance with the Fast Downward planner across a variety of benchmarks. In addition to measuring success rates, we assess how faithfully the generated plans translate into sequences of actions that can actually be executed, identifying both strengths and limitations of using these models in this setting. Our findings show that while the models perform well on simpler planning tasks, they continue to struggle with more complex scenarios that require precise resource management, consistent state tracking, and strict constraint compliance. These results underscore fundamental challenges in applying language models to robotic planning in real world environments. By outlining the gaps that emerge during execution, we aim to guide future research toward combined approaches that integrate language models with classical planners in order to enhance the reliability and scalability of planning in autonomous robotics.

Related papers

• LODGE: Joint Hierarchical Task Planning and Learning of Domain Models with Grounded Execution [16.16223684887115]
  Large Language Models (LLMs) enable planning from natural language instructions using implicit world knowledge.<n>Recent methods aim to learn a problem domain that can be solved for different goal states using classical planners.<n>We address this shortcoming by learning hierarchical domains, where low-level predicates and actions are composed into higher-level counterparts.
  arXiv  Detail & Related papers  (2025-05-15T20:23:21Z)
• LLMs as Planning Modelers: A Survey for Leveraging Large Language Models to Construct Automated Planning Models [24.230622369142193]
  Large Language Models (LLMs) excel in various natural language tasks but often struggle with long-horizon planning problems.<n>This limitation has drawn interest in integrating neuro-symbolic approaches within the Automated Planning (AP) and Natural Language Processing (NLP) communities.
  arXiv  Detail & Related papers  (2025-03-22T03:35:44Z)
• Plan-and-Act: Improving Planning of Agents for Long-Horizon Tasks [36.63527489464188]
  Plan-and-Act is a framework that incorporates explicit planning into large language models (LLMs)<n>Plan-and-Act consists of a Planner model which generates structured, high-level plans to achieve user goals, and an Executor model that translates these plans into environment-specific actions.<n>We present a state-of-the-art 57.58% success rate on the WebArena-Lite benchmark as well as a text-only state-of-the-art 81.36% success rate on WebVoyager.
  arXiv  Detail & Related papers  (2025-03-12T17:40:52Z)
• LangSuitE: Planning, Controlling and Interacting with Large Language Models in Embodied Text Environments [70.91258869156353]
  We introduce LangSuitE, a versatile and simulation-free testbed featuring 6 representative embodied tasks in textual embodied worlds. Compared with previous LLM-based testbeds, LangSuitE offers adaptability to diverse environments without multiple simulation engines. We devise a novel chain-of-thought (CoT) schema, EmMem, which summarizes embodied states w.r.t. history information.
  arXiv  Detail & Related papers  (2024-06-24T03:36:29Z)
• Exploring and Benchmarking the Planning Capabilities of Large Language Models [57.23454975238014]
  This work lays the foundations for improving planning capabilities of large language models (LLMs) We construct a comprehensive benchmark suite encompassing both classical planning benchmarks and natural language scenarios. We investigate the use of many-shot in-context learning to enhance LLM planning, exploring the relationship between increased context length and improved planning performance.
  arXiv  Detail & Related papers  (2024-06-18T22:57:06Z)
• PARADISE: Evaluating Implicit Planning Skills of Language Models with Procedural Warnings and Tips Dataset [0.0]
  We present PARADISE, an abductive reasoning task using Q&A format on practical procedural text sourced from wikiHow. It involves warning and tip inference tasks directly associated with goals, excluding intermediary steps, with the aim of testing the ability of the models to infer implicit knowledge of the plan solely from the given goal. Our experiments, utilizing fine-tuned language models and zero-shot prompting, reveal the effectiveness of task-specific small models over large language models in most scenarios.
  arXiv  Detail & Related papers  (2024-03-05T18:01:59Z)
• What's the Plan? Evaluating and Developing Planning-Aware Techniques for Language Models [7.216683826556268]
  Large language models (LLMs) are increasingly used for applications that require planning capabilities. We introduce SimPlan, a novel hybrid-method, and evaluate its performance in a new challenging setup.
  arXiv  Detail & Related papers  (2024-02-18T07:42:49Z)
• LLM-Assist: Enhancing Closed-Loop Planning with Language-Based Reasoning [65.86754998249224]
  We develop a novel hybrid planner that leverages a conventional rule-based planner in conjunction with an LLM-based planner. Our approach navigates complex scenarios which existing planners struggle with, produces well-reasoned outputs while also remaining grounded through working alongside the rule-based approach.
  arXiv  Detail & Related papers  (2023-12-30T02:53:45Z)
• EgoPlan-Bench: Benchmarking Multimodal Large Language Models for Human-Level Planning [84.6451394629312]
  We introduce EgoPlan-Bench, a benchmark to evaluate the planning abilities of MLLMs in real-world scenarios. We show that EgoPlan-Bench poses significant challenges, highlighting a substantial scope for improvement in MLLMs to achieve human-level task planning. We also present EgoPlan-IT, a specialized instruction-tuning dataset that effectively enhances model performance on EgoPlan-Bench.
  arXiv  Detail & Related papers  (2023-12-11T03:35:58Z)
• Compositional Foundation Models for Hierarchical Planning [52.18904315515153]
  We propose a foundation model which leverages expert foundation model trained on language, vision and action data individually together to solve long-horizon tasks. We use a large language model to construct symbolic plans that are grounded in the environment through a large video diffusion model. Generated video plans are then grounded to visual-motor control, through an inverse dynamics model that infers actions from generated videos.
  arXiv  Detail & Related papers  (2023-09-15T17:44:05Z)
• PlaSma: Making Small Language Models Better Procedural Knowledge Models for (Counterfactual) Planning [77.03847056008598]
  PlaSma is a novel two-pronged approach to endow small language models with procedural knowledge and (constrained) language planning capabilities. We develop symbolic procedural knowledge distillation to enhance the commonsense knowledge in small language models and an inference-time algorithm to facilitate more structured and accurate reasoning.
  arXiv  Detail & Related papers  (2023-05-31T00:55:40Z)
• STRIPS Action Discovery [67.73368413278631]
  Recent approaches have shown the success of classical planning at synthesizing action models even when all intermediate states are missing. We propose a new algorithm to unsupervisedly synthesize STRIPS action models with a classical planner when action signatures are unknown.
  arXiv  Detail & Related papers  (2020-01-30T17:08:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.