PLAY2PROMPT: Zero-shot Tool Instruction Optimization for LLM Agents via Tool Play

• URL: http://arxiv.org/abs/2503.14432v1
• Date: Tue, 18 Mar 2025 17:09:57 GMT
• Title: PLAY2PROMPT: Zero-shot Tool Instruction Optimization for LLM Agents via Tool Play
• Authors: Wei Fang, Yang Zhang, Kaizhi Qian, James Glass, Yada Zhu,
• Abstract summary: Large language models (LLMs) are increasingly integrated with specialized external tools.<n>Many tasks demand zero-shot tool usage with minimal or noisy documentation.<n>We propose PLAY2PROMPT, an automated framework that systematically "plays" with each tool to explore its input-output behaviors.
• Score: 24.784100934155237
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models (LLMs) are increasingly integrated with specialized external tools, yet many tasks demand zero-shot tool usage with minimal or noisy documentation. Existing solutions rely on manual rewriting or labeled data for validation, making them inapplicable in true zero-shot settings. To address these challenges, we propose PLAY2PROMPT, an automated framework that systematically "plays" with each tool to explore its input-output behaviors. Through this iterative trial-and-error process, PLAY2PROMPT refines tool documentation and generates usage examples without any labeled data. These examples not only guide LLM inference but also serve as validation to further enhance tool utilization. Extensive experiments on real-world tasks demonstrate that PLAY2PROMPT significantly improves zero-shot tool performance across both open and closed models, offering a scalable and effective solution for domain-specific tool integration.

Related papers

• Self-Training Large Language Models for Tool-Use Without Demonstrations [15.17750971071501]
  Large language models (LLMs) remain prone to factual inaccuracies and computational errors. Recent work augmented LLMs with tools to mitigate these shortcomings, but often requires curated gold tool-use demonstrations. This paper investigates whether LLMs can learn to use tools without demonstrations.
  arXiv  Detail & Related papers  (2025-02-09T12:06:10Z)
• PTR: Precision-Driven Tool Recommendation for Large Language Models [43.53494041932615]
  We propose a Precision-driven Tool Recommendation (PTR) approach for Large Language Models (LLMs) PTR captures an initial, concise set of tools by leveraging historical tool bundle usage and dynamically adjusts the tool set by performing tool matching. We present a new dataset, RecTools, and a metric, TRACC, designed to evaluate the effectiveness of tool recommendation for LLMs.
  arXiv  Detail & Related papers  (2024-11-14T17:33:36Z)
• Learning to Ask: When LLM Agents Meet Unclear Instruction [55.65312637965779]
  Large language models (LLMs) can leverage external tools for addressing a range of tasks unattainable through language skills alone. We evaluate the performance of LLMs tool-use under imperfect instructions, analyze the error patterns, and build a challenging tool-use benchmark called Noisy ToolBench. We propose a novel framework, Ask-when-Needed (AwN), which prompts LLMs to ask questions to users whenever they encounter obstacles due to unclear instructions.
  arXiv  Detail & Related papers  (2024-08-31T23:06:12Z)
• Tool Learning in the Wild: Empowering Language Models as Automatic Tool Agents [56.822238860147024]
  Augmenting large language models with external tools has emerged as a promising approach to extend their utility.<n>Previous methods manually parse tool documentation and create in-context demonstrations, transforming tools into structured formats for LLMs to use in their step-by-step reasoning.<n>We propose AutoTools, a framework that enables LLMs to automate the tool-use workflow.
  arXiv  Detail & Related papers  (2024-05-26T11:40:58Z)
• Towards Completeness-Oriented Tool Retrieval for Large Language Models [60.733557487886635]
  Real-world systems often incorporate a wide array of tools, making it impractical to input all tools into Large Language Models. Existing tool retrieval methods primarily focus on semantic matching between user queries and tool descriptions. We propose a novel modelagnostic COllaborative Learning-based Tool Retrieval approach, COLT, which captures not only the semantic similarities between user queries and tool descriptions but also takes into account the collaborative information of tools.
  arXiv  Detail & Related papers  (2024-05-25T06:41:23Z)
• Planning, Creation, Usage: Benchmarking LLMs for Comprehensive Tool Utilization in Real-World Complex Scenarios [93.68764280953624]
  UltraTool is a novel benchmark designed to improve and evaluate Large Language Models' ability in tool utilization. It emphasizes real-world complexities, demanding accurate, multi-step planning for effective problem-solving. A key feature of UltraTool is its independent evaluation of planning with natural language, which happens before tool usage.
  arXiv  Detail & Related papers  (2024-01-30T16:52:56Z)
• EASYTOOL: Enhancing LLM-based Agents with Concise Tool Instruction [56.02100384015907]
  EasyTool is a framework transforming diverse and lengthy tool documentation into a unified and concise tool instruction. It can significantly reduce token consumption and improve the performance of tool utilization in real-world scenarios.
  arXiv  Detail & Related papers  (2024-01-11T15:45:11Z)
• Don't Fine-Tune, Decode: Syntax Error-Free Tool Use via Constrained Decoding [11.51687663492722]
  Large language models (LLMs) excel at many tasks but often fail to use external tools due to complicated and unfamiliar syntax constraints. We propose TOOLDEC, a decoding algorithm using finite state machines to force LLMs to follow tool syntax. Experiments show that TOOLDEC eliminates all syntax errors, achieving significantly better performance on various base models and benchmarks.
  arXiv  Detail & Related papers  (2023-10-10T23:37:53Z)
• Tool Documentation Enables Zero-Shot Tool-Usage with Large Language Models [90.96816639172464]
  Large language models (LLMs) are taught to use new tools by providing a few demonstrations of the tool's usage. We advocate the use of tool documentation, descriptions for the individual tool usage, over demonstrations.
  arXiv  Detail & Related papers  (2023-08-01T17:21:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.