Toward Secure and Reliable PDDL Formalization of Large Language Models with Planner-in-the-Loop Feedback

• URL: http://arxiv.org/abs/2606.29700v1
• Date: Mon, 29 Jun 2026 02:08:30 GMT
• Title: Toward Secure and Reliable PDDL Formalization of Large Language Models with Planner-in-the-Loop Feedback
• Abstract summary: Planning often requires symbolic specifications that are both executable and verifiable.<n>We present NL-PDDL-Bench, a benchmark for natural-language-to-PDDL specification construction with planner-verified executability.<n>We also propose a planner-in-the-loop framework that uses validator and planner diagnostics to revise non-executable specifications.
• Score: 11.629045437889943
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Planning often requires symbolic specifications that are both executable and verifiable. For large language models deployed in autonomous or decision-support systems, failures in such formalization may lead to unverifiable decisions, execution failures, or unsafe downstream behavior. We present NL-PDDL-Bench, a multi-domain benchmark for natural-language-to-PDDL specification construction with planner-verified executability and controlled difficulty scaling by object count. We further propose a planner-in-the-loop framework that uses validator and planner diagnostics to revise non-executable specifications through localized edits. Building on this infrastructure, we develop a planner-grounded optimization recipe that combines parameter-efficient Low-Rank Adaptation supervised fine-tuning, offline planner-derived preference pairs for Direct Preference Optimization, and inference-time planner-in-the-loop repair, without requiring online planner calls during training. We also provide a unified evaluation suite for parseability, solvability, specification similarity, and outcome-aware plan-level consistency against planner references. Experiments on representative model families show substantial gains in planner success and plan-level agreement, with improved robustness under difficulty scaling and cross-domain variation. These results highlight the value of externally verifiable formalization for reliable deployment of LLMs in safety- or security-sensitive planning systems. Code and data are available at: https://github.com/ibasicplan/NL-PDDL-Bench

Related papers

• Grounded Evaluation and Repair for NL-to-PDDL Problem Generation [3.323276338973552]
  This paper studies an end-to-end NL-to-PDDL pipeline that combines LLM generation, checks in terms of PDDL parsing, planning and validation, a domain-conformance checker, and iterative repair.<n>Across Planetarium, AutoPlanBench, and curated PDDL2.1 problems, results show that operational success and benchmark-reference reconstruction can substantially diverge.
  arXiv  Detail & Related papers  (2026-09-09T08:53:05Z)
• AdaPlanBench: Evaluating Adaptive Planning in Large Language Model Agents under World and User Constraints [65.27107190916512]
  We introduce AdaPlanBench, an interactive benchmark for evaluating whether Large Language Model (LLM) agents can adaptively plan and re-plan under progressively revealed constraints.<n>AdaPlanBench is built on 307 household tasks, with a scalable constraint construction pipeline that augments each task with dual constraints.<n>Experiments show that adaptive planning under dual constraints remains challenging, with the best model reaching only 67.75% accuracy.
  arXiv  Detail & Related papers  (2026-06-04T02:47:29Z)
• DisasterBench: Benchmarking LLM Planning under Typed Tool Interface Constraints [29.573427850164222]
  Disasters cause severe societal impacts, demanding rapid coordination of heterogeneous AI tools.<n>Effective coordination requires more than selecting semantically plausible tools.<n>We introduce DisasterBench, a benchmark for evaluating structured multi-agent planning.
  arXiv  Detail & Related papers  (2026-05-27T04:50:23Z)
• PlanningBench: Generating Scalable and Verifiable Planning Data for Evaluating and Training Large Language Models [52.48858778580074]
  Planning is a fundamental capability for large language models (LLMs)<n>PlanningBench is a framework for generating scalable, diverse, verifiable planning data for both evaluation and training.
  arXiv  Detail & Related papers  (2026-05-20T08:10:15Z)
• PILOT: Planning via Internalized Latent Optimization Trajectories for Large Language Models [51.43746425777865]
  Large Language Models (LLMs) often lack the capacity to formulate global strategies, leading to error propagation in long-horizon tasks.<n>We propose PILOT, a framework designed to internalize the strategic oversight of large models into intrinsic Latent Guidance.
  arXiv  Detail & Related papers  (2026-01-07T12:38:56Z)
• An End-to-end Planning Framework with Agentic LLMs and PDDL [9.718390674899771]
  We present an end-to-end framework for planning supported by verifiers.<n>An orchestrator receives a human specification written in natural language and converts it into a PDDL model.<n>The validated domain and problem are then passed to an external planning engine to generate a plan.
  arXiv  Detail & Related papers  (2025-12-10T13:17:08Z)
• SDA-PLANNER: State-Dependency Aware Adaptive Planner for Embodied Task Planning [22.01842981739722]
  Embodied task planning requires agents to produce executable actions in a close-loop manner within the environment.<n>We propose SDA-PLANNER, enabling an adaptive planning paradigm, state-dependency aware and error-aware mechanisms for comprehensive embodied task planning.
  arXiv  Detail & Related papers  (2025-09-30T15:07:59Z)
• Teaching LLMs to Plan: Logical Chain-of-Thought Instruction Tuning for Symbolic Planning [23.185497225384207]
  Large language models (LLMs) have demonstrated impressive capabilities across diverse tasks, yet their ability to perform structured symbolic planning remains limited.<n>We present a novel instruction tuning framework, PDDL-Instruct, designed to enhance LLMs' symbolic planning capabilities through logical chain-of-thought reasoning.
  arXiv  Detail & Related papers  (2025-09-14T02:42:34Z)
• Interactive and Expressive Code-Augmented Planning with Large Language Models [62.799579304821826]
  Large Language Models (LLMs) demonstrate strong abilities in common-sense reasoning and interactive decision-making. Recent techniques have sought to structure LLM outputs using control flow and other code-adjacent techniques to improve planning performance. We propose REPL-Plan, an LLM planning approach that is fully code-expressive and dynamic.
  arXiv  Detail & Related papers  (2024-11-21T04:23:17Z)
• Unlocking Reasoning Potential in Large Langauge Models by Scaling Code-form Planning [94.76546523689113]
  We introduce CodePlan, a framework that generates and follows textcode-form plans -- pseudocode that outlines high-level, structured reasoning processes. CodePlan effectively captures the rich semantics and control flows inherent to sophisticated reasoning tasks. It achieves a 25.1% relative improvement compared with directly generating responses.
  arXiv  Detail & Related papers  (2024-09-19T04:13:58Z)
• AdaPlanner: Adaptive Planning from Feedback with Language Models [56.367020818139665]
  Large language models (LLMs) have recently demonstrated the potential in acting as autonomous agents for sequential decision-making tasks. We propose a closed-loop approach, AdaPlanner, which allows the LLM agent to refine its self-generated plan adaptively in response to environmental feedback. To mitigate hallucination, we develop a code-style LLM prompt structure that facilitates plan generation across a variety of tasks, environments, and agent capabilities.
  arXiv  Detail & Related papers  (2023-05-26T05:52:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.