Documentation Retrieval Improves Planning Language Generation

• URL: http://arxiv.org/abs/2509.19931v2
• Date: Mon, 29 Sep 2025 06:20:52 GMT
• Title: Documentation Retrieval Improves Planning Language Generation
• Abstract summary: We present a series of lightweight pipelines to improve open-source planning models.<n>Our best pipeline improves plan correctness from 0% to over 80% on the common BlocksWorld domain.<n>However, semantic errors persist in more challenging domains, revealing fundamental limitations in current models' reasoning capabilities.
• Score: 3.90727941420584
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Certain strong LLMs have shown promise for zero-shot formal planning by generating planning languages like PDDL. Yet, the performance of most open-source models under 50B parameters has been reported to be close to zero due to the low-resource nature of these languages. We significantly improve their performance via a series of lightweight pipelines that integrates documentation retrieval with modular code generation and error refinement. With models like Llama-4-Maverick, our best pipeline improves plan correctness from 0% to over 80% on the common BlocksWorld domain. However, while syntactic errors are substantially reduced, semantic errors persist in more challenging domains, revealing fundamental limitations in current models' reasoning capabilities.

Related papers

• ZAS-SQL: Distilling Rules from Failures for Zero-Shot Text-to-SQL [4.477529152940812]
  Few-shot in-context learning methods built upon large language models (LLMs) achieve strong performance, yet their reliance on demonstrations limits cross-domain generalization.<n>We propose a fully zero-shot Text-to-Guided framework that distills core generation rules from failure cases through a Map-Reduce-based rule distillation pipeline.<n>The proposed framework achieves up to 87.2% and 88.6% execution accuracy on the Dev and Test sets, respectively, establishing a new zero-shot state-of-the-art.
  arXiv  Detail & Related papers  (2026-06-06T16:08:40Z)
• Learn2Zinc: Fine-tuning Small Language Models for Text-to-Model Translation in MiniZinc [3.9124823111588163]
  We investigate whether targeted fine-tuning can teach small language models to generate correct MiniZinc models from natural language problem descriptions.<n>Key finding is that syntax errors dominate failures when working with this domain specific language.<n>We propose a cross-model error bootstrapping approach that collects syntax errors from multiple LLM runs and leverage those to curate an error correction training dataset.
  arXiv  Detail & Related papers  (2026-05-14T22:55:14Z)
• Anka: A Domain-Specific Language for Reliable LLM Code Generation [0.0]
  Large Language Models (LLMs) exhibit systematic errors on complex, multi-step programming tasks.<n>We introduce Anka, a domain-specific language () for data transformation pipelines designed with explicit, constrained syntax.<n>Anka achieves 99.9% parse success and 95.8% overall task accuracy across 100 benchmark problems.
  arXiv  Detail & Related papers  (2025-12-29T05:28:17Z)
• Evaluating the Limitations of Local LLMs in Solving Complex Programming Challenges [0.31498833540989407]
  This study examines the performance of open-source, locally hosted large-language models (LLMs) in handling complex programming tasks.<n>Building on the original Framework for AI-driven Code Generation Evaluation (FACE), the authors retrofit the pipeline to work entirely offline.<n>Results show that the overall pass@1 accuracy is modest for the local models, with the best models performing at approximately half the acceptance rate of the proprietary models.
  arXiv  Detail & Related papers  (2025-09-18T14:13:30Z)
• Addressing the Challenges of Planning Language Generation [6.209697341255856]
  We design and evaluate 8 different PDDL generation pipelines with open-source models under 50 billion parameters.<n>We find that intuitive approaches such as using a high-resource language wrapper or constrained decoding with grammar decrease performance, yet inference-time scaling approaches such as revision with feedback from the solver and plan validator more than double the performance.
  arXiv  Detail & Related papers  (2025-05-20T17:25:23Z)
• Syntactic and Semantic Control of Large Language Models via Sequential Monte Carlo [90.78001821963008]
  A wide range of LM applications require generating text that conforms to syntactic or semantic constraints.<n>We develop an architecture for controlled LM generation based on sequential Monte Carlo (SMC)<n>Our system builds on the framework of Lew et al. (2023) and integrates with its language model probabilistic programming language.
  arXiv  Detail & Related papers  (2025-04-17T17:49:40Z)
• Type-Constrained Code Generation with Language Models [51.03439021895432]
  We introduce a type-constrained decoding approach that leverages type systems to guide code generation.<n>For this purpose, we develop novel prefix automata and a search over inhabitable types, forming a sound approach to enforce well-typedness on LLM-generated code.<n>Our approach reduces compilation errors by more than half and significantly increases functional correctness in code synthesis, translation, and repair tasks.
  arXiv  Detail & Related papers  (2025-04-12T15:03:00Z)
• Enhancing Code Generation for Low-Resource Languages: No Silver Bullet [55.39571645315926]
  Large Language Models (LLMs) rely on large and diverse datasets to learn syntax, semantics, and usage patterns of programming languages.<n>For low-resource languages, the limited availability of such data hampers the models' ability to generalize effectively.<n>We present an empirical study investigating the effectiveness of several approaches for boosting LLMs' performance on low-resource languages.
  arXiv  Detail & Related papers  (2025-01-31T12:23:28Z)
• Challenges in Adapting Multilingual LLMs to Low-Resource Languages using LoRA PEFT Tuning [0.4194295877935868]
  This study investigates the effects of Low-Rank Adaptation (LoRA) -Efficient Fine-Tuning (PEFT) on multilingual Gemma models for Marathi.<n>Using a translated dataset with 52,000 instruction-response pairs, our findings reveal that while evaluation performance decline post-fine-tuning, manual assessments frequently suggest that the fine-tuned models outperform their original counterparts.
  arXiv  Detail & Related papers  (2024-11-27T18:14:38Z)
• GALLa: Graph Aligned Large Language Models for Improved Source Code Understanding [35.536478348255315]
  Recent code language models have scaled to billions of parameters, but model source code solely as text tokens.<n>We take the best of both worlds with GALLa - Graph Aligned Large Language Models.
  arXiv  Detail & Related papers  (2024-09-06T10:57:34Z)
• Unlocking the Potential of Model Merging for Low-Resource Languages [66.7716891808697]
  Adapting large language models to new languages typically involves continual pre-training (CT) followed by supervised fine-tuning (SFT) We propose model merging as an alternative for low-resource languages, combining models with distinct capabilities into a single model without additional training. Experiments based on Llama-2-7B demonstrate that model merging effectively endows LLMs for low-resource languages with task-solving abilities, outperforming CT-then-SFT in scenarios with extremely scarce data.
  arXiv  Detail & Related papers  (2024-07-04T15:14:17Z)
• Multi-Reference Preference Optimization for Large Language Models [56.84730239046117]
  We introduce a novel closed-form formulation for direct preference optimization using multiple reference models. The resulting algorithm, Multi-Reference Preference Optimization (MRPO), leverages broader prior knowledge from diverse reference models. Our experiments demonstrate that LLMs finetuned with MRPO generalize better in various preference data, regardless of data scarcity or abundance.
  arXiv  Detail & Related papers  (2024-05-26T00:29:04Z)
• GlotLID: Language Identification for Low-Resource Languages [51.38634652914054]
  GlotLID-M is an LID model that satisfies the desiderata of wide coverage, reliability and efficiency. It identifies 1665 languages, a large increase in coverage compared to prior work.
  arXiv  Detail & Related papers  (2023-10-24T23:45:57Z)
• Leveraging Pre-trained Large Language Models to Construct and Utilize World Models for Model-based Task Planning [39.29964085305846]
  Methods that use pre-trained large language models directly as planners are currently impractical due to limited correctness of plans. In this work, we introduce a novel alternative paradigm that constructs an explicit world (domain) model in planning domain definition language (PDDL) and then uses it to plan with sound domain-independent planners.
  arXiv  Detail & Related papers  (2023-05-24T08:59:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.