What We Talk About When We Talk About LLM Planning: Evidence for Two Distinct Planning Abilities

• URL: http://arxiv.org/abs/2607.11197v1
• Date: Mon, 13 Jul 2026 07:47:17 GMT
• Title: What We Talk About When We Talk About LLM Planning: Evidence for Two Distinct Planning Abilities
• Abstract summary: We argue that task-level variation may reflect distinct latent planning competencies rather than differences along a single ability spectrum.<n>We study this question on ACPBench-Hard by evaluating multiple LLM families under varying test-time reasoning budgets.
• Score: 43.87329561965196
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: When LLMs exhibit uneven performance across planning tasks, these gaps are often attributed to task difficulty. We argue that this explanation is incomplete, as task-level variation may reflect distinct latent planning competencies rather than differences along a single ability spectrum. We study this question on ACPBench-Hard by evaluating multiple LLM families under varying test-time reasoning budgets and applying a multidimensional item response theory model to uncover the latent competency structure underlying LLM planning. The analysis reveals two principal dimensions that shape planning performance: operational reasoning, the ability to evaluate local action applicability and immediate state transitions, and structural enumeration, the ability to reason about goal reachability and landmark structure. Operational reasoning improving under model scaling and longer reasoning traces, while structural enumeration remains comparatively insensitive. Our findings motivate competency-level evaluation of LLM planning, shifting the focus from whether models improve overall to which planning competencies improve, under what conditions, and why.

Related papers

• Knowledge Model Prompting Increases LLM Performance on Planning Tasks [0.0]
  This paper investigates whether the Task-Method-Knowledge framework can improve large language models' reasoning capabilities.<n>The study evaluates TMK by experimenting on the PlanBench benchmark to test for reasoning and planning capabilities.
  arXiv  Detail & Related papers  (2026-02-03T09:47:05Z)
• Teaching LLMs to Plan: Logical Chain-of-Thought Instruction Tuning for Symbolic Planning [23.185497225384207]
  Large language models (LLMs) have demonstrated impressive capabilities across diverse tasks, yet their ability to perform structured symbolic planning remains limited.<n>We present a novel instruction tuning framework, PDDL-Instruct, designed to enhance LLMs' symbolic planning capabilities through logical chain-of-thought reasoning.
  arXiv  Detail & Related papers  (2025-09-14T02:42:34Z)
• PLAN-TUNING: Post-Training Language Models to Learn Step-by-Step Planning for Complex Problem Solving [66.42260489147617]
  We introduce PLAN-TUNING, a framework that distills synthetic task decompositions from large-scale language models.<n>Plan-TUNING fine-tunes smaller models via supervised and reinforcement-learning objectives to improve complex reasoning.<n>Our analysis demonstrates how planning trajectories improves complex reasoning capabilities.
  arXiv  Detail & Related papers  (2025-07-10T07:30:44Z)
• On the Eligibility of LLMs for Counterfactual Reasoning: A Decompositional Study [15.617243755155686]
  Counterfactual reasoning has emerged as a crucial technique for generalizing the reasoning capabilities of large language models.<n>We propose a decompositional strategy that breaks down the counterfactual generation from causality construction to the reasoning over counterfactual interventions.
  arXiv  Detail & Related papers  (2025-05-17T04:59:32Z)
• HyperTree Planning: Enhancing LLM Reasoning via Hierarchical Thinking [109.09735490692202]
  We propose HyperTree Planning (HTP), a novel reasoning paradigm that constructs hypertree-structured planning outlines for effective planning.<n> Experiments demonstrate the effectiveness of HTP, achieving state-of-the-art accuracy on the TravelPlanner benchmark with Gemini-1.5-Pro, resulting in a 3.6 times performance improvement over o1-preview.
  arXiv  Detail & Related papers  (2025-05-05T02:38:58Z)
• Deliberate Reasoning in Language Models as Structure-Aware Planning with an Accurate World Model [14.480267340831542]
  Structure-aware Planning with an Accurate World Model (SWAP)<n>SWAP integrates structured knowledge representation with learned planning.<n>We evaluate SWAP across diverse reasoning-intensive benchmarks including math reasoning, logical reasoning, and coding tasks.
  arXiv  Detail & Related papers  (2024-10-04T04:23:36Z)
• On The Planning Abilities of OpenAI's o1 Models: Feasibility, Optimality, and Generalizability [59.72892401927283]
  We evaluate the planning capabilities of OpenAI's o1 models across a variety of benchmark tasks. Our results reveal that o1-preview outperforms GPT-4 in adhering to task constraints.
  arXiv  Detail & Related papers  (2024-09-30T03:58:43Z)
• VSP: Assessing the dual challenges of perception and reasoning in spatial planning tasks for VLMs [102.36953558562436]
  Vision language models (VLMs) are an exciting emerging class of language models (LMs) One understudied capability inVLMs is visual spatial planning. Our study introduces a benchmark that evaluates the spatial planning capability in these models in general.
  arXiv  Detail & Related papers  (2024-07-02T00:24:01Z)
• Exploring and Benchmarking the Planning Capabilities of Large Language Models [57.23454975238014]
  This work lays the foundations for improving planning capabilities of large language models (LLMs) We construct a comprehensive benchmark suite encompassing both classical planning benchmarks and natural language scenarios. We investigate the use of many-shot in-context learning to enhance LLM planning, exploring the relationship between increased context length and improved planning performance.
  arXiv  Detail & Related papers  (2024-06-18T22:57:06Z)
• PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Change [34.93870615625937]
  PlanBench is a benchmark suite based on the kinds of domains used in the automated planning community. PlanBench provides sufficient diversity in both the task domains and the specific planning capabilities.
  arXiv  Detail & Related papers  (2022-06-21T16:15:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.