ToolCoder: Teach Code Generation Models to use API search tools
- URL: http://arxiv.org/abs/2305.04032v5
- Date: Mon, 11 Sep 2023 06:33:46 GMT
- Title: ToolCoder: Teach Code Generation Models to use API search tools
- Authors: Kechi Zhang, Huangzhao Zhang, Ge Li, Jia Li, Zhuo Li, Zhi Jin
- Abstract summary: We propose ToolCoder, a novel approach that integrates API search tools with existing models to assist in code generation and API selection.
Our experimental results demonstrate that ToolCoder exhibits excellent performance and generalization across five public and private library code generation benchmarks.
- Score: 44.370920906850024
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Automatically generating source code from natural language descriptions has
been a growing field of research in recent years. However, current large-scale
code generation models often encounter difficulties when selecting appropriate
APIs for specific contexts. These models may generate APIs that do not meet
requirements or refer to non-existent APIs in third-party libraries, especially
for lesser-known or private libraries. Inspired by the process of human
developers using tools to search APIs, we propose ToolCoder, a novel approach
that integrates API search tools with existing models to assist in code
generation and API selection. To teach our model to use tools, we introduce an
automated data annotation method using ChatGPT to add tool usage information
into the source code data and fine-tune code generation models. During
inference, we integrate API search tools into the generation process so that
our model can automatically use the search tool to get suggestions when
selecting an API. Our experimental results demonstrate that ToolCoder exhibits
excellent performance and generalization across five public and private library
code generation benchmarks, with at least 6.21\% improvement on average pass@1
metrics and 9.64\% improvement on average pass@10 metrics compared to
state-of-the-art methods. Furthermore, we show that our relatively small
ToolCoder model is comparable to one of the current best models, GPT-3.5,
highlighting the potential of incorporating programming tools into the code
generation process.
Related papers
- A Comprehensive Framework for Evaluating API-oriented Code Generation in Large Language Models [14.665460257371164]
Large language models (LLMs) like GitHub Copilot and ChatGPT have emerged as powerful tools for code generation.
We propose AutoAPIEval, a framework designed to evaluate the capabilities of LLMs in API-oriented code generation.
arXiv Detail & Related papers (2024-09-23T17:22:09Z) - Efficient and Scalable Estimation of Tool Representations in Vector Space [34.767193045989515]
We present a framework for generating synthetic data for tool retrieval applications and an efficient data-driven tool retrieval strategy using small encoder models.
We create ToolBank, a new tool retrieval dataset that reflects real human user usages.
With these new methods, we achieve improvements of up to 27.28 in Recall@K on the ToolBench dataset and 30.5 in Recall@K on ToolBank.
arXiv Detail & Related papers (2024-09-02T19:39:24Z) - FANTAstic SEquences and Where to Find Them: Faithful and Efficient API Call Generation through State-tracked Constrained Decoding and Reranking [57.53742155914176]
API call generation is the cornerstone of large language models' tool-using ability.
Existing supervised and in-context learning approaches suffer from high training costs, poor data efficiency, and generated API calls that can be unfaithful to the API documentation and the user's request.
We propose an output-side optimization approach called FANTASE to address these limitations.
arXiv Detail & Related papers (2024-07-18T23:44:02Z) - Are Human Rules Necessary? Generating Reusable APIs with CoT Reasoning and In-Context Learning [14.351476383642016]
We propose a novel approach, named Code2API, to automatically perform APIzation for Stack Overflow code snippets.
Code2API does not require additional model training or any manual crafting rules.
It can be easily deployed on personal computers without relying on other external tools.
arXiv Detail & Related papers (2024-05-06T14:22:17Z) - ModelScope-Agent: Building Your Customizable Agent System with
Open-source Large Language Models [74.64651681052628]
We introduce ModelScope-Agent, a customizable agent framework for real-world applications based on open-source LLMs as controllers.
It provides a user-friendly system library, with customizable engine design to support model training on multiple open-source LLMs.
A comprehensive framework has been proposed spanning over tool-use data collection, tool retrieval, tool registration, memory control, customized model training, and evaluation.
arXiv Detail & Related papers (2023-09-02T16:50:30Z) - ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world
APIs [104.37772295581088]
Open-source large language models (LLMs), e.g., LLaMA, remain significantly limited in tool-use capabilities.
We introduce ToolLLM, a general tool-usetuning encompassing data construction, model training, and evaluation.
We first present ToolBench, an instruction-tuning framework for tool use, which is constructed automatically using ChatGPT.
arXiv Detail & Related papers (2023-07-31T15:56:53Z) - Private-Library-Oriented Code Generation with Large Language Models [52.73999698194344]
This paper focuses on utilizing large language models (LLMs) for code generation in private libraries.
We propose a novel framework that emulates the process of programmers writing private code.
We create four private library benchmarks, including TorchDataEval, TorchDataComplexEval, MonkeyEval, and BeatNumEval.
arXiv Detail & Related papers (2023-07-28T07:43:13Z) - Toolformer: Language Models Can Teach Themselves to Use Tools [62.04867424598204]
Language models (LMs) exhibit remarkable abilities to solve new tasks from just a few examples or textual instructions, especially at scale.
We show that LMs can teach themselves to use external tools via simple APIs and achieve the best of both worlds.
We introduce Toolformer, a model trained to decide which APIs to call, when to call them, what arguments to pass, and how to best incorporate the results into future token prediction.
arXiv Detail & Related papers (2023-02-09T16:49:57Z) - On the Effectiveness of Pretrained Models for API Learning [8.788509467038743]
Developers frequently use APIs to implement certain functionalities, such as parsing Excel Files, reading and writing text files line by line, etc.
Developers can greatly benefit from automatic API usage sequence generation based on natural language queries for building applications in a faster and cleaner manner.
Existing approaches utilize information retrieval models to search for matching API sequences given a query or use RNN-based encoder-decoder to generate API sequences.
arXiv Detail & Related papers (2022-04-05T20:33:24Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.