Grounded Evaluation and Repair for NL-to-PDDL Problem Generation

• URL: http://arxiv.org/abs/2609.09898v1
• Date: Wed, 09 Sep 2026 08:53:05 GMT
• Title: Grounded Evaluation and Repair for NL-to-PDDL Problem Generation
• Abstract summary: This paper studies an end-to-end NL-to-PDDL pipeline that combines LLM generation, checks in terms of PDDL parsing, planning and validation, a domain-conformance checker, and iterative repair.<n>Across Planetarium, AutoPlanBench, and curated PDDL2.1 problems, results show that operational success and benchmark-reference reconstruction can substantially diverge.
• Score: 3.323276338973552
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large Language Models (LLMs) have shown promise for translating Natural Language (NL) planning descriptions into PDDL problem instances. However, standard evaluation criteria such as syntactic validity or planner success can substantially overestimate faithfulness to the described task: a generated problem may be parseable and solvable while misrepresenting the intended initial state, goal, object structure, or optimization target. This paper studies an end-to-end NL-to-PDDL pipeline that combines LLM generation, checks in terms of PDDL parsing, planning and validation, a domain-conformance checker, an LLM critic, and iterative repair. Fine-grained repair feedback is constructed from the domain description, the generated problem, the natural language problem description, and operational diagnostics. Reference-based comparisons against curated benchmark PDDL problem descriptions are used for post-hoc benchmark analysis, and these offline checks include renaming-invariant structural matching and semantic equivalence, where domain support is available. Across Planetarium, AutoPlanBench, and curated PDDL~2.1 problems, results show that operational success and benchmark-reference reconstruction can diverge substantially. Results also show that structured repair can be useful, and that PDDL~2.1 remains challenging for reference reconstruction, even when operational success improves.

Related papers

• CAST: Critique-Aware Supervision for Training Reliable Long-Horizon Tool-Calling Agents [66.9575676266637]
  Large language model (LLM) agents are increasingly deployed in long-horizon, interactive, and stateful environments.<n>In these settings, a single wrong action, such as refunding the wrong purchase, can cause irreversible task failure and must be intercepted before execution.<n>We develop CAST, a critique-aware training framework that converts sparse task outcomes into action-level supervision for critique learning and policy optimization.
  arXiv  Detail & Related papers  (2026-08-31T01:59:51Z)
• RepoReasoner: Evaluating Repository-Level Code Reasoning Ability of Long-Context Language Models [54.53236295237077]
  We introduce RepoReasoner, a benchmark for evaluating repository-level code reasoning.<n> Output Prediction measures fine-grained, stateful execution reasoning across files.<n>Call Chain Prediction evaluates high-level architectural dependency understanding under noisy context.
  arXiv  Detail & Related papers  (2026-07-28T17:12:41Z)
• Toward Secure and Reliable PDDL Formalization of Large Language Models with Planner-in-the-Loop Feedback [11.629045437889943]
  Planning often requires symbolic specifications that are both executable and verifiable.<n>We present NL-PDDL-Bench, a benchmark for natural-language-to-PDDL specification construction with planner-verified executability.<n>We also propose a planner-in-the-loop framework that uses validator and planner diagnostics to revise non-executable specifications.
  arXiv  Detail & Related papers  (2026-06-29T02:08:30Z)
• Self-Evolving Spatial Reasoning in Vision Language Models via Geometric Logic Consistency [53.28076739140119]
  We propose a framework that enforces logical consistency in Vision-Language Models (VLMs) through geometric and linguistic duality operations.<n>SAGE is model-agnostic, data-efficient compared to prior GRPO methods, and can be applied as a lightweight post-training stage to any existing VLM.<n> Experiments on video and spatial reasoning benchmarks demonstrate consistent improvements over strong baselines.
  arXiv  Detail & Related papers  (2026-05-18T10:05:21Z)
• Structured Prompting and Feedback-Guided Reasoning with LLMs for Data Interpretation [0.0]
  Large language models (LLMs) have demonstrated remarkable capabilities in natural language understanding and task generalization.<n>This paper introduces the STROT Framework, a method for structured prompting and feedback-driven transformation logic generation.
  arXiv  Detail & Related papers  (2025-05-03T00:05:01Z)
• Generating Symbolic World Models via Test-time Scaling of Large Language Models [28.258707611580643]
  Planning Domain Definition Language (PDDL) is leveraged as a planning abstraction that enables precise and formal state descriptions.<n>We introduce a simple yet effective algorithm, which first employs a Best-of-N sampling approach to improve the quality of the initial solution.<n>Our method outperforms o1-mini by a considerable margin in the generation of PDDL domains.
  arXiv  Detail & Related papers  (2025-02-07T07:52:25Z)
• Automated Refactoring of Non-Idiomatic Python Code: A Differentiated Replication with LLMs [54.309127753635366]
  We present the results of a replication study in which we investigate GPT-4 effectiveness in recommending and suggesting idiomatic actions.<n>Our findings underscore the potential of LLMs to achieve tasks where, in the past, implementing recommenders based on complex code analyses was required.
  arXiv  Detail & Related papers  (2025-01-28T15:41:54Z)
• Self-Evolving Critique Abilities in Large Language Models [59.861013614500024]
  This paper explores enhancing critique abilities of Large Language Models (LLMs)<n>We introduce SCRIT, a framework that trains LLMs with self-generated data to evolve their critique abilities.<n>Our analysis reveals that SCRIT's performance scales positively with data and model size.
  arXiv  Detail & Related papers  (2025-01-10T05:51:52Z)
• In-context Demonstration Matters: On Prompt Optimization for Pseudo-Supervision Refinement [71.60563181678323]
  Large language models (LLMs) have achieved great success across diverse tasks, and fine-tuning is sometimes needed to further enhance generation quality.<n>To handle these challenges, a direct solution is to generate high-confidence'' data from unsupervised downstream tasks.<n>We propose a novel approach, pseudo-supervised demonstrations aligned prompt optimization (PAPO) algorithm, which jointly refines both the prompt and the overall pseudo-supervision.
  arXiv  Detail & Related papers  (2024-10-04T03:39:28Z)
• Planetarium: A Rigorous Benchmark for Translating Text to Structured Planning Languages [20.62336315814875]
  We introduce textitPlanetarium, a benchmark designed to evaluate language models' ability to generate PDDL code from natural language descriptions of planning tasks.<n>textitPlanetarium features a novel PDDL equivalence algorithm that flexibly evaluates the correctness of generated PDDL, along with a dataset of 145,918 text-to-PDDL pairs.
  arXiv  Detail & Related papers  (2024-07-03T17:59:53Z)
• Generating consistent PDDL domains with Large Language Models [4.8551773468225745]
  Large Language Models (LLMs) are capable of transforming natural language domain descriptions into plausibly looking PDDL markup. We present a novel concept to significantly improve the quality of LLM-generated PDDL models by performing automated consistency checking during the generation process. Although the proposed consistency checking strategies still can't guarantee absolute correctness of generated models, they can serve as valuable source of feedback reducing the amount of correction efforts expected from a human in the loop.
  arXiv  Detail & Related papers  (2024-04-11T13:48:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.