Contextual API Completion for Unseen Repositories Using LLMs

• URL: http://arxiv.org/abs/2405.04600v3
• Date: Tue, 14 May 2024 21:50:29 GMT
• Title: Contextual API Completion for Unseen Repositories Using LLMs
• Authors: Noor Nashid, Taha Shabani, Parsa Alian, Ali Mesbah,
• Abstract summary: We introduce a novel technique to mitigate hallucinations by leveraging global and local contextual information within a code repository for API completion tasks. Our approach is tailored to refine code completion tasks, with a focus on optimizing local API completions. Our tool, LANCE, surpasses Copilot by 143% and 142% for API token completion and conversational API completion, respectively.
• Score: 6.518508607788089
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models have made substantial progress in addressing diverse code-related tasks. However, their adoption is hindered by inconsistencies in generating output due to the lack of real-world, domain-specific information, such as for intra-repository API calls for unseen software projects. We introduce a novel technique to mitigate hallucinations by leveraging global and local contextual information within a code repository for API completion tasks. Our approach is tailored to refine code completion tasks, with a focus on optimizing local API completions. We examine relevant import statements during API completion to derive insights into local APIs, drawing from their method signatures. For API token completion, we analyze the inline variables and correlate them with the appropriate imported modules, thereby allowing our approach to rank the most contextually relevant suggestions from the available local APIs. Further, for conversational API completion, we gather APIs that are most relevant to the developer query with a retrieval-based search across the project. We employ our tool, LANCE, within the framework of our proposed benchmark, APIEval, encompassing two different programming languages. Our evaluation yields an average accuracy of 82.6% for API token completion and 76.9% for conversational API completion tasks. On average, LANCE surpasses Copilot by 143% and 142% for API token completion and conversational API completion, respectively. The implications of our findings are substantial for developers, suggesting that our lightweight context analysis can be applied to multilingual environments without language-specific training or fine-tuning, allowing for efficient implementation with minimal examples and effort.

Related papers

• A Systematic Evaluation of Large Code Models in API Suggestion: When, Which, and How [53.65636914757381]
  API suggestion is a critical task in modern software development. Recent advancements in large code models (LCMs) have shown promise in the API suggestion task.
  arXiv  Detail & Related papers  (2024-09-20T03:12:35Z)
• FANTAstic SEquences and Where to Find Them: Faithful and Efficient API Call Generation through State-tracked Constrained Decoding and Reranking [57.53742155914176]
  API call generation is the cornerstone of large language models' tool-using ability. Existing supervised and in-context learning approaches suffer from high training costs, poor data efficiency, and generated API calls that can be unfaithful to the API documentation and the user's request. We propose an output-side optimization approach called FANTASE to address these limitations.
  arXiv  Detail & Related papers  (2024-07-18T23:44:02Z)
• WorldAPIs: The World Is Worth How Many APIs? A Thought Experiment [49.00213183302225]
  We propose a framework to induce new APIs by grounding wikiHow instruction to situated agent policies. Inspired by recent successes in large language models (LLMs) for embodied planning, we propose a few-shot prompting to steer GPT-4.
  arXiv  Detail & Related papers  (2024-07-10T15:52:44Z)
• A Solution-based LLM API-using Methodology for Academic Information Seeking [49.096714812902576]
  SoAy is a solution-based LLM API-using methodology for academic information seeking. It uses code with a solution as the reasoning method, where a solution is a pre-constructed API calling sequence. Results show a 34.58-75.99% performance improvement compared to state-of-the-art LLM API-based baselines.
  arXiv  Detail & Related papers  (2024-05-24T02:44:14Z)
• LLM+Reasoning+Planning for supporting incomplete user queries in presence of APIs [0.09374652839580183]
  In practice, natural language task requests (user queries) are often incomplete, i.e., they may not contain all the information required by the APIs. We leverage logical reasoning and classical AI planning along with an LLM for accurately answering user queries. Our approach achieves over 95% success rate in most cases on a dataset containing complete and incomplete single goal and multi-goal queries.
  arXiv  Detail & Related papers  (2024-05-21T01:16:34Z)
• APICom: Automatic API Completion via Prompt Learning and Adversarial Training-based Data Augmentation [6.029137544885093]
  API recommendation is the process of assisting developers in finding the required API among numerous candidate APIs. Previous studies mainly modeled API recommendation as the recommendation task, and developers may not yet be able to find what they need. Motivated by the neural machine translation research domain, we can model this problem as the generation task. We propose a novel approach APICom based on prompt learning, which can generate API related to the query according to the prompts.
  arXiv  Detail & Related papers  (2023-09-13T15:31:50Z)
• Enhancing API Documentation through BERTopic Modeling and Summarization [0.0]
  This paper focuses on the complexities of interpreting Application Programming Interface (API) documentation. Official API documentation serves as a primary source of information for developers, but it can often be extensive and lacks user-friendliness. Our novel approach employs the strengths of BERTopic for topic modeling and Natural Language Processing (NLP) to automatically generate summaries of API documentation.
  arXiv  Detail & Related papers  (2023-08-17T15:57:12Z)
• Evaluating Embedding APIs for Information Retrieval [51.24236853841468]
  We evaluate the capabilities of existing semantic embedding APIs on domain generalization and multilingual retrieval. We find that re-ranking BM25 results using the APIs is a budget-friendly approach and is most effective in English. For non-English retrieval, re-ranking still improves the results, but a hybrid model with BM25 works best, albeit at a higher cost.
  arXiv  Detail & Related papers  (2023-05-10T16:40:52Z)
• Binding Language Models in Symbolic Languages [146.3027328556881]
  Binder is a training-free neural-symbolic framework that maps the task input to a program. In the parsing stage, Codex is able to identify the part of the task input that cannot be answerable by the original programming language. In the execution stage, Codex can perform versatile functionalities given proper prompts in the API calls.
  arXiv  Detail & Related papers  (2022-10-06T12:55:17Z)
• On the Effectiveness of Pretrained Models for API Learning [8.788509467038743]
  Developers frequently use APIs to implement certain functionalities, such as parsing Excel Files, reading and writing text files line by line, etc. Developers can greatly benefit from automatic API usage sequence generation based on natural language queries for building applications in a faster and cleaner manner. Existing approaches utilize information retrieval models to search for matching API sequences given a query or use RNN-based encoder-decoder to generate API sequences.
  arXiv  Detail & Related papers  (2022-04-05T20:33:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.