Identifying and Mitigating API Misuse in Large Language Models

• URL: http://arxiv.org/abs/2503.22821v1
• Date: Fri, 28 Mar 2025 18:43:12 GMT
• Title: Identifying and Mitigating API Misuse in Large Language Models
• Authors: Terry Yue Zhuo, Junda He, Jiamou Sun, Zhenchang Xing, David Lo, John Grundy, Xiaoning Du,
• Abstract summary: API misuse in code generated by large language models (LLMs) represents a serious emerging challenge in software development.<n>This paper presents the first comprehensive study of API misuse patterns in LLM-generated code, analyzing both method selection and parameter usage across Python and Java.<n>We propose Dr.Fix, a novel LLM-based automatic program repair approach for API misuse based on the aforementioned taxonomy.
• Score: 26.4403427473915
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: API misuse in code generated by large language models (LLMs) represents a serious emerging challenge in software development. While LLMs have demonstrated impressive code generation capabilities, their interactions with complex library APIs remain highly prone to errors, potentially leading to software failures and security vulnerabilities. This paper presents the first comprehensive study of API misuse patterns in LLM-generated code, analyzing both method selection and parameter usage across Python and Java. Through extensive manual annotation of 3,892 method-level and 2,560 parameter-level misuses, we develop a novel taxonomy of four distinct API misuse types specific to LLMs, which significantly differ from traditional human-centric misuse patterns. Our evaluation of two widely used LLMs, StarCoder-7B (open-source) and Copilot (closed-source), reveals significant challenges in API usage, particularly in areas of hallucination and intent misalignment. We propose Dr.Fix, a novel LLM-based automatic program repair approach for API misuse based on the aforementioned taxonomy. Our method substantially improves repair accuracy for real-world API misuse, demonstrated by increases of up to 38.4 points in BLEU scores and 40 percentage points in exact match rates across different models and programming languages. This work provides crucial insights into the limitations of current LLMs in API usage and presents an effective solution for the automated repair of API misuse in LLM-generated code.

Related papers

• APIRAT: Integrating Multi-source API Knowledge for Enhanced Code Translation with LLMs [6.522570957351905]
  APIRAT is a novel code translation method that integrates multi-source API knowledge. APIRAT employs three API knowledge augmentation techniques, including API sequence retrieval, API sequence back-translation, and API mapping. Experiments indicate that APIRAT significantly surpasses existing LLM-based methods, achieving improvements in computational accuracy ranging from 4% to 15.1%.
  arXiv  Detail & Related papers  (2025-04-21T04:24:49Z)
• Your Fix Is My Exploit: Enabling Comprehensive DL Library API Fuzzing with Large Language Models [49.214291813478695]
  Deep learning (DL) libraries, widely used in AI applications, often contain vulnerabilities like overflows and use buffer-free errors.<n>Traditional fuzzing struggles with the complexity and API diversity of DL libraries.<n>We propose DFUZZ, an LLM-driven fuzzing approach for DL libraries.
  arXiv  Detail & Related papers  (2025-01-08T07:07:22Z)
• ExploraCoder: Advancing code generation for multiple unseen APIs via planning and chained exploration [70.26807758443675]
  ExploraCoder is a training-free framework that empowers large language models to invoke unseen APIs in code solution.<n>We show that ExploraCoder significantly improves performance for models lacking prior API knowledge, achieving an absolute increase of 11.24% over niave RAG approaches and 14.07% over pretraining methods in pass@10.
  arXiv  Detail & Related papers  (2024-12-06T19:00:15Z)
• APILOT: Navigating Large Language Models to Generate Secure Code by Sidestepping Outdated API Pitfalls [15.865915079829943]
  APILOT maintains a realtime, quickly updatable dataset of outdated APIs. It uses an augmented generation method to navigate LLMs in generating secure, version-aware code. It can reduce outdated code recommendations by 89.42% on average with limited performance overhead.
  arXiv  Detail & Related papers  (2024-09-25T00:37:40Z)
• A Comprehensive Framework for Evaluating API-oriented Code Generation in Large Language Models [14.665460257371164]
  Large language models (LLMs) like GitHub Copilot and ChatGPT have emerged as powerful tools for code generation. We propose AutoAPIEval, a framework designed to evaluate the capabilities of LLMs in API-oriented code generation.
  arXiv  Detail & Related papers  (2024-09-23T17:22:09Z)
• Harnessing LLMs for API Interactions: A Framework for Classification and Synthetic Data Generation [0.0]
  We propose a novel system that integrates Large Language Models (LLMs) for both classifying natural language inputs into corresponding API calls. Our system allows users to invoke complex software functionalities through simple inputs, improving interaction efficiency and lowering the barrier to software utilization.
  arXiv  Detail & Related papers  (2024-09-18T04:56:52Z)
• Exploring Automatic Cryptographic API Misuse Detection in the Era of LLMs [60.32717556756674]
  This paper introduces a systematic evaluation framework to assess Large Language Models in detecting cryptographic misuses. Our in-depth analysis of 11,940 LLM-generated reports highlights that the inherent instabilities in LLMs can lead to over half of the reports being false positives. The optimized approach achieves a remarkable detection rate of nearly 90%, surpassing traditional methods and uncovering previously unknown misuses in established benchmarks.
  arXiv  Detail & Related papers  (2024-07-23T15:31:26Z)
• What's Wrong with Your Code Generated by Large Language Models? An Extensive Study [80.18342600996601]
  Large language models (LLMs) produce code that is shorter yet more complicated as compared to canonical solutions. We develop a taxonomy of bugs for incorrect codes that includes three categories and 12 sub-categories, and analyze the root cause for common bug types. We propose a novel training-free iterative method that introduces self-critique, enabling LLMs to critique and correct their generated code based on bug types and compiler feedback.
  arXiv  Detail & Related papers  (2024-07-08T17:27:17Z)
• Octopus: On-device language model for function calling of software APIs [9.78611123915888]
  Large Language Models (LLMs) play a crucial role due to their advanced text processing and generation abilities. This study introduces a new strategy aimed at harnessing on-device LLMs in invoking software APIs.
  arXiv  Detail & Related papers  (2024-04-02T01:29:28Z)
• LM-Polygraph: Uncertainty Estimation for Language Models [71.21409522341482]
  Uncertainty estimation (UE) methods are one path to safer, more responsible, and more effective use of large language models (LLMs) We introduce LM-Polygraph, a framework with implementations of a battery of state-of-the-art UE methods for LLMs in text generation tasks, with unified program interfaces in Python. It introduces an extendable benchmark for consistent evaluation of UE techniques by researchers, and a demo web application that enriches the standard chat dialog with confidence scores.
  arXiv  Detail & Related papers  (2023-11-13T15:08:59Z)
• ReEval: Automatic Hallucination Evaluation for Retrieval-Augmented Large Language Models via Transferable Adversarial Attacks [91.55895047448249]
  This paper presents ReEval, an LLM-based framework using prompt chaining to perturb the original evidence for generating new test cases. We implement ReEval using ChatGPT and evaluate the resulting variants of two popular open-domain QA datasets. Our generated data is human-readable and useful to trigger hallucination in large language models.
  arXiv  Detail & Related papers  (2023-10-19T06:37:32Z)
• Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedback [127.75419038610455]
  Large language models (LLMs) are able to generate human-like, fluent responses for many downstream tasks. This paper proposes a LLM-Augmenter system, which augments a black-box LLM with a set of plug-and-play modules.
  arXiv  Detail & Related papers  (2023-02-24T18:48:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.