Tool Learning with Foundation Models
- URL: http://arxiv.org/abs/2304.08354v2
- Date: Thu, 15 Jun 2023 14:10:42 GMT
- Title: Tool Learning with Foundation Models
- Authors: Yujia Qin, Shengding Hu, Yankai Lin, Weize Chen, Ning Ding, Ganqu Cui,
Zheni Zeng, Yufei Huang, Chaojun Xiao, Chi Han, Yi Ren Fung, Yusheng Su,
Huadong Wang, Cheng Qian, Runchu Tian, Kunlun Zhu, Shihao Liang, Xingyu Shen,
Bokai Xu, Zhen Zhang, Yining Ye, Bowen Li, Ziwei Tang, Jing Yi, Yuzhang Zhu,
Zhenning Dai, Lan Yan, Xin Cong, Yaxi Lu, Weilin Zhao, Yuxiang Huang, Junxi
Yan, Xu Han, Xian Sun, Dahai Li, Jason Phang, Cheng Yang, Tongshuang Wu, Heng
Ji, Zhiyuan Liu, Maosong Sun
- Abstract summary: With the advent of foundation models, AI systems have the potential to be equally adept in tool use as humans.
Despite its immense potential, there is still a lack of a comprehensive understanding of key challenges, opportunities, and future endeavors in this field.
- Score: 114.2581831746077
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Humans possess an extraordinary ability to create and utilize tools, allowing
them to overcome physical limitations and explore new frontiers. With the
advent of foundation models, AI systems have the potential to be equally adept
in tool use as humans. This paradigm, i.e., tool learning with foundation
models, combines the strengths of specialized tools and foundation models to
achieve enhanced accuracy, efficiency, and automation in problem-solving.
Despite its immense potential, there is still a lack of a comprehensive
understanding of key challenges, opportunities, and future endeavors in this
field. To this end, we present a systematic investigation of tool learning in
this paper. We first introduce the background of tool learning, including its
cognitive origins, the paradigm shift of foundation models, and the
complementary roles of tools and models. Then we recapitulate existing tool
learning research into tool-augmented and tool-oriented learning. We formulate
a general tool learning framework: starting from understanding the user
instruction, models should learn to decompose a complex task into several
subtasks, dynamically adjust their plan through reasoning, and effectively
conquer each sub-task by selecting appropriate tools. We also discuss how to
train models for improved tool-use capabilities and facilitate the
generalization in tool learning. Considering the lack of a systematic tool
learning evaluation in prior works, we experiment with 18 representative tools
and show the potential of current foundation models in skillfully utilizing
tools. Finally, we discuss several open problems that require further
investigation for tool learning. Overall, we hope this paper could inspire
future research in integrating tools with foundation models.
Related papers
- MetaTool: Facilitating Large Language Models to Master Tools with Meta-task Augmentation [25.360660222418183]
We introduce a new tool learning methodology (MetaTool) that is generalizable for mastering any reusable toolset.
We develop a series of meta-tasks that involve predicting masked factors of tool execution.
By incorporating meta-task data into the instruction tuning process, the proposed MetaTool model achieves significant superiority to open-source models.
arXiv Detail & Related papers (2024-07-15T10:15:41Z) - Tool-Planner: Dynamic Solution Tree Planning for Large Language Model with Tool Clustering [30.25234781338571]
We propose Tool-Planner, a task-processing framework based on toolkits.
Tool-Planner groups tools based on the API functions with the same function into a toolkit.
When a tool error occurs, the language model can reselect and adjust tools based on the toolkit.
arXiv Detail & Related papers (2024-06-06T07:30:14Z) - Tool Learning with Large Language Models: A Survey [60.733557487886635]
Tool learning with large language models (LLMs) has emerged as a promising paradigm for augmenting the capabilities of LLMs to tackle highly complex problems.
Despite growing attention and rapid advancements in this field, the existing literature remains fragmented and lacks systematic organization.
arXiv Detail & Related papers (2024-05-28T08:01:26Z) - Chain of Tools: Large Language Model is an Automatic Multi-tool Learner [54.992464510992605]
Automatic Tool Chain (ATC) is a framework that enables the large language models (LLMs) to act as a multi-tool user.
To scale up the scope of the tools, we next propose a black-box probing method.
For a comprehensive evaluation, we build a challenging benchmark named ToolFlow.
arXiv Detail & Related papers (2024-05-26T11:40:58Z) - COLT: Towards Completeness-Oriented Tool Retrieval for Large Language Models [60.733557487886635]
We propose a novel modelagnostic COllaborative Learning-based Tool Retrieval approach, COLT.
COLT captures semantic similarities between user queries and tool descriptions.
It also takes into account the collaborative information of tools.
arXiv Detail & Related papers (2024-05-25T06:41:23Z) - ToolEyes: Fine-Grained Evaluation for Tool Learning Capabilities of
Large Language Models in Real-world Scenarios [48.38419686697733]
We propose ToolEyes, a fine-grained system tailored for the evaluation of large language models' tool learning capabilities in authentic scenarios.
The system meticulously examines seven real-world scenarios, analyzing five dimensions crucial to LLMs in tool learning.
ToolEyes incorporates a tool library boasting approximately 600 tools, serving as an intermediary between LLMs and the physical world.
arXiv Detail & Related papers (2024-01-01T12:49:36Z) - Learning Generalizable Tool-use Skills through Trajectory Generation [13.879860388944214]
We train a single model on four different deformable object manipulation tasks.
The model generalizes to various novel tools, significantly outperforming baselines.
We further test our trained policy in the real world with unseen tools, where it achieves the performance comparable to human.
arXiv Detail & Related papers (2023-09-29T21:32:42Z) - Large Language Models as Tool Makers [85.00361145117293]
We introduce a closed-loop framework, referred to as LLMs A s Tool Makers (LATM), where LLMs create their own reusable tools for problem-solving.
Our approach consists of two phases: 1) tool making: an LLM acts as the tool maker that crafts tools for a set of tasks. 2) tool using: another LLM acts as the tool user, which applies the tool built by the tool maker for problem-solving.
arXiv Detail & Related papers (2023-05-26T17:50:11Z) - Making Language Models Better Tool Learners with Execution Feedback [36.30542737293863]
Tools serve as pivotal interfaces that enable humans to understand and reshape the environment.
Existing tool learning methodologies induce large language models to utilize tools indiscriminately.
We propose Tool leaRning wIth exeCution fEedback (TRICE), a two-stage end-to-end framework that enables the model to continually learn through feedback derived from tool execution.
arXiv Detail & Related papers (2023-05-22T14:37:05Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.