Benchmarking Web API Integration Code Generation

• URL: http://arxiv.org/abs/2509.20172v4
• Date: Sun, 09 Nov 2025 19:22:28 GMT
• Title: Benchmarking Web API Integration Code Generation
• Authors: Daniel Maninger, Leon Chemnitz, Amir Molzam Sharifloo, Jannis Brugger, Mira Mezini,
• Abstract summary: We present WAPIIBench, a dataset and evaluation pipeline designed to assess the ability of LLMs to generate web API invocation code.<n>Our experiments with several open-source LLMs reveal that generating API invocations poses a significant challenge.<n>None of the evaluated open-source models was able to solve more than 40% of the tasks.
• Score: 3.5669873833301047
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: API integration is a cornerstone of our digital infrastructure, enabling software systems to connect and interact. However, as shown by many studies, writing or generating correct code to invoke APIs, particularly web APIs, is challenging. Although large language models (LLMs) have become popular in software development, their effectiveness in automating the generation of web API integration code remains unexplored. In order to address this, we present WAPIIBench, a dataset and evaluation pipeline designed to assess the ability of LLMs to generate web API invocation code. Our experiments with several open-source LLMs reveal that generating API invocations poses a significant challenge, resulting in hallucinated endpoints, incorrect argument usage, and other errors. None of the evaluated open-source models was able to solve more than 40% of the tasks.

Related papers

• Framework-Aware Code Generation with API Knowledge Graph-Constructed Data: A Study on HarmonyOS [52.483888557864326]
  APIKG4SYN is a framework designed to exploit API knowledge graphs for the construction of API-oriented question-code pairs.<n>We build the first benchmark for HarmonyOS code generation using APIKG4SYN.
  arXiv  Detail & Related papers  (2025-11-29T08:13:54Z)
• Your Fix Is My Exploit: Enabling Comprehensive DL Library API Fuzzing with Large Language Models [49.214291813478695]
  Deep learning (DL) libraries, widely used in AI applications, often contain vulnerabilities like overflows and use buffer-free errors.<n>Traditional fuzzing struggles with the complexity and API diversity of DL libraries.<n>We propose DFUZZ, an LLM-driven fuzzing approach for DL libraries.
  arXiv  Detail & Related papers  (2025-01-08T07:07:22Z)
• ExploraCoder: Advancing code generation for multiple unseen APIs via planning and chained exploration [70.26807758443675]
  ExploraCoder is a training-free framework that empowers large language models to invoke unseen APIs in code solution.<n> Experimental results demonstrate that ExploraCoder significantly improves performance for models lacking prior API knowledge.
  arXiv  Detail & Related papers  (2024-12-06T19:00:15Z)
• Demystifying Application Programming Interfaces (APIs): Unlocking the Power of Large Language Models and Other Web-based AI Services in Social Work Research [0.0]
  Application Programming Interfaces (APIs) are essential tools for social work researchers aiming to harness advanced technologies like Large Language Models (LLMs) and other AI services. This paper demystifies APIs and illustrates how they can enhance research methodologies. Practical code examples demonstrate how LLMs can generate API code for accessing specialized services, such as extracting data from unstructured text.
  arXiv  Detail & Related papers  (2024-10-26T16:07:12Z)
• A Comprehensive Framework for Evaluating API-oriented Code Generation in Large Language Models [14.665460257371164]
  Large language models (LLMs) like GitHub Copilot and ChatGPT have emerged as powerful tools for code generation. We propose AutoAPIEval, a framework designed to evaluate the capabilities of LLMs in API-oriented code generation.
  arXiv  Detail & Related papers  (2024-09-23T17:22:09Z)
• FANTAstic SEquences and Where to Find Them: Faithful and Efficient API Call Generation through State-tracked Constrained Decoding and Reranking [57.53742155914176]
  API call generation is the cornerstone of large language models' tool-using ability. Existing supervised and in-context learning approaches suffer from high training costs, poor data efficiency, and generated API calls that can be unfaithful to the API documentation and the user's request. We propose an output-side optimization approach called FANTASE to address these limitations.
  arXiv  Detail & Related papers  (2024-07-18T23:44:02Z)
• SoAy: A Solution-based LLM API-using Methodology for Academic Information Seeking [59.59923482238048]
  SoAy is a solution-based LLM API-using methodology for academic information seeking.<n>It uses code with a solution as the reasoning method, where a solution is a pre-constructed API calling sequence.<n>Results show a 34.58-75.99% performance improvement compared to state-of-the-art LLM API-based baselines.
  arXiv  Detail & Related papers  (2024-05-24T02:44:14Z)
• Octopus: On-device language model for function calling of software APIs [9.78611123915888]
  Large Language Models (LLMs) play a crucial role due to their advanced text processing and generation abilities. This study introduces a new strategy aimed at harnessing on-device LLMs in invoking software APIs.
  arXiv  Detail & Related papers  (2024-04-02T01:29:28Z)
• SpeCrawler: Generating OpenAPI Specifications from API Documentation Using Large Language Models [8.372941103284774]
  SpeCrawler is a comprehensive system that generates OpenAPI Specifications from diverse API documentation. The paper explores SpeCrawler's methodology, supported by empirical evidence and case studies.
  arXiv  Detail & Related papers  (2024-02-18T15:33:24Z)
• Private-Library-Oriented Code Generation with Large Language Models [52.73999698194344]
  This paper focuses on utilizing large language models (LLMs) for code generation in private libraries. We propose a novel framework that emulates the process of programmers writing private code. We create four private library benchmarks, including TorchDataEval, TorchDataComplexEval, MonkeyEval, and BeatNumEval.
  arXiv  Detail & Related papers  (2023-07-28T07:43:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.