Agent Planning Benchmark: A Diagnostic Framework for Planning Capabilities in LLM Agents

• URL: http://arxiv.org/abs/2606.04874v2
• Date: Fri, 05 Jun 2026 05:23:52 GMT
• Title: Agent Planning Benchmark: A Diagnostic Framework for Planning Capabilities in LLM Agents
• Abstract summary: We introduce Agent Planning Benchmark (APB), a planning-specific diagnostic benchmark with 4,209 multimodal cases across 22 domains and five settings.<n>APB reveals systematic weaknesses in long-horizon planning, tool-noise robustness, calibrated refusal, and inference-time refinement.<n>APB serves as an upstream diagnostic complement to execution benchmarks.
• Score: 40.17728515217378
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Planning is central to LLM agents: before acting, an agent must decompose goals, select tools, reason over constraints, and decide when a task is infeasible. Yet existing agent evaluations often report only end-to-end success, making it difficult to determine whether failures stem from planning or execution. We introduce Agent Planning Benchmark (APB), a planning-specific diagnostic benchmark with 4,209 multimodal cases across 22 domains and five settings, covering holistic planning, feedback-conditioned step-wise planning, and robustness under extraneous tools, broken tools, and unsolvable tasks. Across 12 MLLMs, APB reveals systematic weaknesses in long-horizon planning, tool-noise robustness, calibrated refusal, and inference-time refinement. We further validate APB on 200 ToolSandbox tasks and 200 $τ^2$-bench tasks, where APB-guided refinement consistently improves plan correctness, plan grade, and downstream execution metrics across three representative models. APB thus serves as an upstream diagnostic complement to execution benchmarks. The APB benchmark and code are available in \href{https://github.com/Mikivishy/AgentPlanningBenchmark}{this URL}.

Related papers

• PlanBench-XL: Evaluating Long-Horizon Planning of LLM Tool-Use Agents in Large-Scale Tool Ecosystems [59.730861364166174]
  PlanBench-XL is an interactive benchmark of 327 retail tasks over 1,665 tools.<n>It tests whether agents can iteratively retrieve usable tools, invoke them to uncover intermediate evidence for subsequent calls toward the final goal.
  arXiv  Detail & Related papers  (2026-06-21T08:29:12Z)
• PlanningBench: Generating Scalable and Verifiable Planning Data for Evaluating and Training Large Language Models [52.48858778580074]
  Planning is a fundamental capability for large language models (LLMs)<n>PlanningBench is a framework for generating scalable, diverse, verifiable planning data for both evaluation and training.
  arXiv  Detail & Related papers  (2026-05-20T08:10:15Z)
• From Plan to Action: How Well Do Agents Follow the Plan? [6.54753809716901]
  This paper presents the first systematic analysis of plan compliance in programming agents.<n>We examined 16,991 trajectories from SWE-agents across four LLMs on SWE-bench Verified and SWEbench Pro under eight plan variations.<n>A subpar plan hurts performance even more than no plan at all.
  arXiv  Detail & Related papers  (2026-04-13T23:54:55Z)
• AI Planning Framework for LLM-Based Web Agents [2.9376953730570197]
  We introduce a taxonomy that maps modern agent architectures to traditional planning paradigms.<n>We propose five novel evaluation metrics that assess trajectory quality beyond simple success rates.<n>Our results show that while the Step-by-Step agent aligns more closely with human gold trajectories, the Full-Plan-in-Advance agent excels in technical measures.
  arXiv  Detail & Related papers  (2026-03-13T06:46:32Z)
• What Is Your Agent's GPA? A Framework for Evaluating Agent Goal-Plan-Action Alignment [3.5583478152586756]
  Agent GPA is an evaluation paradigm based on an agent's operational loop of setting goals, devising plans, and executing actions.<n>The framework includes five evaluation metrics: Goal Fulfillment, Logical Consistency, Execution Efficiency, Plan Quality, and Plan Adherence.
  arXiv  Detail & Related papers  (2025-10-09T22:40:19Z)
• Hindsight Planner: A Closed-Loop Few-Shot Planner for Embodied Instruction Following [62.10809033451526]
  This work focuses on building a task planner for Embodied Instruction Following (EIF) using Large Language Models (LLMs)<n>We frame the task as a Partially Observable Markov Decision Process (POMDP) and aim to develop a robust planner under a few-shot assumption.<n>Our experiments on the ALFRED dataset indicate that our planner achieves competitive performance under a few-shot assumption.
  arXiv  Detail & Related papers  (2024-12-27T10:05:45Z)
• Ask-before-Plan: Proactive Language Agents for Real-World Planning [68.08024918064503]
  Proactive Agent Planning requires language agents to predict clarification needs based on user-agent conversation and agent-environment interaction. We propose a novel multi-agent framework, Clarification-Execution-Planning (textttCEP), which consists of three agents specialized in clarification, execution, and planning.
  arXiv  Detail & Related papers  (2024-06-18T14:07:28Z)
• AdaPlanner: Adaptive Planning from Feedback with Language Models [56.367020818139665]
  Large language models (LLMs) have recently demonstrated the potential in acting as autonomous agents for sequential decision-making tasks. We propose a closed-loop approach, AdaPlanner, which allows the LLM agent to refine its self-generated plan adaptively in response to environmental feedback. To mitigate hallucination, we develop a code-style LLM prompt structure that facilitates plan generation across a variety of tasks, environments, and agent capabilities.
  arXiv  Detail & Related papers  (2023-05-26T05:52:27Z)
• A Framework for Neurosymbolic Robot Action Planning using Large Language Models [3.0501524254444767]
  We present a framework aimed at bridging the gap between symbolic task planning and machine learning approaches. The rationale is training Large Language Models (LLMs) into a neurosymbolic task planner compatible with the Planning Domain Definition Language (PDDL) Preliminary results in selected domains show that our method can: (i) solve 95.5% of problems in a test data set of 1,000 samples; (ii) produce plans up to 13.5% shorter than a traditional symbolic planner; (iii) reduce average overall waiting times for a plan availability by up to 61.4%.
  arXiv  Detail & Related papers  (2023-03-01T11:54:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.