CRAFT: Customizing LLMs by Creating and Retrieving from Specialized
Toolsets
- URL: http://arxiv.org/abs/2309.17428v2
- Date: Wed, 13 Mar 2024 05:39:25 GMT
- Title: CRAFT: Customizing LLMs by Creating and Retrieving from Specialized
Toolsets
- Authors: Lifan Yuan, Yangyi Chen, Xingyao Wang, Yi R. Fung, Hao Peng, Heng Ji
- Abstract summary: We present CRAFT, a tool creation and retrieval framework for large language models (LLMs)
It creates toolsets specifically curated for the tasks and equips LLMs with a component that retrieves tools from these sets to enhance their capability to solve complex tasks.
Our method is designed to be flexible and offers a plug-and-play approach to adapt off-the-shelf LLMs to unseen domains and modalities, without any finetuning.
- Score: 75.64181719386497
- License: http://creativecommons.org/licenses/by-nc-sa/4.0/
- Abstract: Large language models (LLMs) are often augmented with tools to solve complex
tasks. By generating code snippets and executing them through task-specific
Application Programming Interfaces (APIs), they can offload certain functions
to dedicated external modules, such as image encoding and performing
calculations. However, most existing approaches to augment LLMs with tools are
constrained by general-purpose APIs and lack the flexibility for tailoring them
to specific tasks. In this work, we present CRAFT, a general tool creation and
retrieval framework for LLMs. It creates toolsets specifically curated for the
tasks and equips LLMs with a component that retrieves tools from these sets to
enhance their capability to solve complex tasks. For each task, we collect
specific code solutions by prompting GPT-4 to solve the training examples.
Following a validation step ensuring the correctness, these solutions are
abstracted into code snippets to enhance reusability, and deduplicated for
higher quality. At inference time, the language model retrieves snippets from
the toolsets and then executes them or generates the output conditioning on the
retrieved snippets. Our method is designed to be flexible and offers a
plug-and-play approach to adapt off-the-shelf LLMs to unseen domains and
modalities, without any finetuning. Experiments on vision-language, tabular
processing, and mathematical reasoning tasks show that our approach achieves
substantial improvements compared to strong baselines. In addition, our
in-depth analysis reveals that: (1) consistent performance improvement can be
achieved by scaling up the number of tools and the capability of the backbone
models; (2) each component of our approach contributes to the performance
gains; (3) the created tools are well-structured and reliable with low
complexity and atomicity. The code is available at
https://github.com/lifan-yuan/CRAFT.
Related papers
- StepTool: A Step-grained Reinforcement Learning Framework for Tool Learning in LLMs [44.906714156993694]
We introduce StepTool, a novel step-grained reinforcement learning framework to improve tool learning in Large Language Models.
StepTool significantly outperforms existing methods in multi-step, tool-based tasks.
arXiv Detail & Related papers (2024-10-10T09:23:26Z) - LLM With Tools: A Survey [0.0]
This paper delves into the methodology,challenges, and developments in the realm of teaching LLMs to use external tools.
We introduce a standardized paradigm for tool integration guided by a series of functions that map user instructions to actionable plans.
Our exploration reveals the various challenges encountered, such as tool invocation timing, selection accuracy, and the need for robust reasoning processes.
arXiv Detail & Related papers (2024-09-24T14:08:11Z) - Chain of Tools: Large Language Model is an Automatic Multi-tool Learner [54.992464510992605]
Automatic Tool Chain (ATC) is a framework that enables the large language models (LLMs) to act as a multi-tool user.
To scale up the scope of the tools, we next propose a black-box probing method.
For a comprehensive evaluation, we build a challenging benchmark named ToolFlow.
arXiv Detail & Related papers (2024-05-26T11:40:58Z) - Efficient Tool Use with Chain-of-Abstraction Reasoning [65.18096363216574]
Large language models (LLMs) need to ground their reasoning to real-world knowledge.
There remains challenges for fine-tuning LLM agents to invoke tools in multi-step reasoning problems.
We propose a new method for LLMs to better leverage tools in multi-step reasoning.
arXiv Detail & Related papers (2024-01-30T21:53:30Z) - Small LLMs Are Weak Tool Learners: A Multi-LLM Agent [73.54562551341454]
Large Language Model (LLM) agents significantly extend the capabilities of standalone LLMs.
We propose a novel approach that decomposes the aforementioned capabilities into a planner, caller, and summarizer.
This modular framework facilitates individual updates and the potential use of smaller LLMs for building each capability.
arXiv Detail & Related papers (2024-01-14T16:17:07Z) - Large Language Models as Tool Makers [85.00361145117293]
We introduce a closed-loop framework, referred to as LLMs A s Tool Makers (LATM), where LLMs create their own reusable tools for problem-solving.
Our approach consists of two phases: 1) tool making: an LLM acts as the tool maker that crafts tools for a set of tasks. 2) tool using: another LLM acts as the tool user, which applies the tool built by the tool maker for problem-solving.
arXiv Detail & Related papers (2023-05-26T17:50:11Z) - CREATOR: Tool Creation for Disentangling Abstract and Concrete Reasoning of Large Language Models [74.22729793816451]
Large Language Models (LLMs) have made significant progress in utilizing tools, but their ability is limited by API availability.
We propose CREATOR, a novel framework that enables LLMs to create their own tools using documentation and code realization.
We evaluate CREATOR on MATH and TabMWP benchmarks, respectively consisting of challenging math competition problems.
arXiv Detail & Related papers (2023-05-23T17:51:52Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.