Look Before You Leap: Towards Decision-Aware and Generalizable
Tool-Usage for Large Language Models
- URL: http://arxiv.org/abs/2402.16696v2
- Date: Wed, 28 Feb 2024 03:31:28 GMT
- Title: Look Before You Leap: Towards Decision-Aware and Generalizable
Tool-Usage for Large Language Models
- Authors: Anchun Gui, Jian Li, Yong Dai, Nan Du, Han Xiao
- Abstract summary: We propose a decision-aware and generalizable tool-usage framework (DEER)
Specifically, we first construct the tool-usage samples with multiple decision branches via an automatic generation pipeline.
Our proposed DEER is effective and significantly outperforms baselines across various datasets.
- Score: 28.19932548630398
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Tool-augmented large language models (LLMs) are attracting widespread
attention when accessing up-to-date knowledge and alleviating hallucination
issues. Nowadays, advanced closed-source LLMs (e.g., ChatGPT) have demonstrated
surprising tool-usage capabilities through prompting and in-context learning
techniques. To empower the capabilities of open-source LLMs (e.g., LLaMA) in
manipulating tools, current efforts focus on either template-driven or
token-triggered tool-usage. However, the former hampers LLMs' flexibility to
address diverse user's queries due to constrained tool interactions, while the
latter limits the generalizability when engaging with new tools, since
tool-usage learning is based on task- and tool-specific datasets. To alleviate
these concerns, in this paper, we propose a decision-aware and generalizable
tool-usage framework (DEER). Specifically, we first construct the tool-usage
samples with multiple decision branches via an automatic generation pipeline,
thereby inspiring the decision-making awareness of LLMs under diverse
scenarios. Meanwhile, we propose a novel tool sampling strategy to enhance the
generalizability of LLMs over unseen tools. Extensive experiments demonstrate
that our proposed DEER is effective and significantly outperforms baselines
across various datasets.
Related papers
- Chain of Tools: Large Language Model is an Automatic Multi-tool Learner [54.992464510992605]
Automatic Tool Chain (ATC) is a framework that enables the large language models (LLMs) to act as a multi-tool user.
To scale up the scope of the tools, we next propose a black-box probing method.
For a comprehensive evaluation, we build a challenging benchmark named ToolFlow.
arXiv Detail & Related papers (2024-05-26T11:40:58Z) - COLT: Towards Completeness-Oriented Tool Retrieval for Large Language Models [60.733557487886635]
We propose a novel modelagnostic COllaborative Learning-based Tool Retrieval approach, COLT.
COLT captures semantic similarities between user queries and tool descriptions.
It also takes into account the collaborative information of tools.
arXiv Detail & Related papers (2024-05-25T06:41:23Z) - Planning, Creation, Usage: Benchmarking LLMs for Comprehensive Tool Utilization in Real-World Complex Scenarios [93.68764280953624]
UltraTool is a novel benchmark designed to improve and evaluate Large Language Models' ability in tool utilization.
It emphasizes real-world complexities, demanding accurate, multi-step planning for effective problem-solving.
A key feature of UltraTool is its independent evaluation of planning with natural language, which happens before tool usage.
arXiv Detail & Related papers (2024-01-30T16:52:56Z) - EASYTOOL: Enhancing LLM-based Agents with Concise Tool Instruction [56.02100384015907]
EasyTool is a framework transforming diverse and lengthy tool documentation into a unified and concise tool instruction.
It can significantly reduce token consumption and improve the performance of tool utilization in real-world scenarios.
arXiv Detail & Related papers (2024-01-11T15:45:11Z) - MetaTool Benchmark for Large Language Models: Deciding Whether to Use
Tools and Which to Use [82.24774504584066]
Large language models (LLMs) have garnered significant attention due to their impressive natural language processing (NLP) capabilities.
We introduce MetaTool, a benchmark designed to evaluate whether LLMs have tool usage awareness and can correctly choose tools.
We conduct experiments involving eight popular LLMs and find that the majority of them still struggle to effectively select tools.
arXiv Detail & Related papers (2023-10-04T19:39:26Z) - On the Tool Manipulation Capability of Open-source Large Language Models [19.6917640220883]
We show can we enhance open-source LLMs to be competitive to leading closed LLM APIs in tool manipulation.
Our techniques can boost leading open-source LLMs by up to 90% success rate, showing capabilities competitive to OpenAI GPT-4 in 4 out of 8 ToolBench tasks.
arXiv Detail & Related papers (2023-05-25T22:10:20Z) - CREATOR: Tool Creation for Disentangling Abstract and Concrete Reasoning of Large Language Models [74.22729793816451]
Large Language Models (LLMs) have made significant progress in utilizing tools, but their ability is limited by API availability.
We propose CREATOR, a novel framework that enables LLMs to create their own tools using documentation and code realization.
We evaluate CREATOR on MATH and TabMWP benchmarks, respectively consisting of challenging math competition problems.
arXiv Detail & Related papers (2023-05-23T17:51:52Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.