LLM-assisted Mutation for Whitebox API Testing

• URL: http://arxiv.org/abs/2504.05738v1
• Date: Tue, 08 Apr 2025 07:14:51 GMT
• Title: LLM-assisted Mutation for Whitebox API Testing
• Authors: Jia Li, Jiacheng Shen, Yuxin Su, Michael R. Lyu,
• Abstract summary: MioHint is a novel white-box API testing approach that leverages the code comprehension capabilities of Large Language Model (LLM) to boost API testing.<n>To evaluate the effectiveness of our method, we conducted experiments across 16 real-world API services.
• Score: 40.91007243855959
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Cloud applications heavily rely on APIs to communicate with each other and exchange data. To ensure the reliability of cloud applications, cloud providers widely adopt API testing techniques. Unfortunately, existing API testing approaches are insufficient to reach strict conditions, a problem known as fitness plateaus, due to the lack of gradient provided by coverage metrics. To address this issue, we propose MioHint, a novel white-box API testing approach that leverages the code comprehension capabilities of Large Language Model (LLM) to boost API testing. The key challenge of LLM-based API testing lies in system-level testing, which emphasizes the dependencies between requests and targets across functions and files, thereby making the entire codebase the object of analysis. However, feeding the entire codebase to an LLM is impractical due to its limited context length and short memory. MioHint addresses this challenge by synergizing static analysis with LLMs. We retrieve relevant code with data-dependency analysis at the statement level, including def-use analysis for variables used in the target and function expansion for subfunctions called by the target. To evaluate the effectiveness of our method, we conducted experiments across 16 real-world REST API services. The findings reveal that MioHint achieves an average increase of 4.95% absolute in line coverage compared to the baseline, EvoMaster, alongside a remarkable factor of 67x improvement in mutation accuracy. Furthermore, our method successfully covers over 57% of hard-to-cover targets while in baseline the coverage is less than 10%.

Related papers

• Combining Static and Dynamic Approaches for Mining and Testing Constraints for RESTful API Testing [8.972346309150199]
  We propose to combine a novel static analysis approach (in which the constraints for API response bodies are mined from API specifications) with the dynamic approach. We leverage large language models (LLMs) to comprehend the API specifications, mine constraints for response bodies, and generate test cases. We also use its generated test cases to detect 21 mismatches between the API specification and actual response data for 8 real-world APIs.
  arXiv  Detail & Related papers  (2025-04-24T06:28:18Z)
• Test Amplification for REST APIs via Single and Multi-Agent LLM Systems [1.6499388997661122]
  We show how single-agent and multi-agent LLM systems can amplify a REST API test suite. Our evaluation demonstrates increased API coverage, identification of numerous bugs in the API under test, and insights into the computational cost and energy consumption of both approaches.
  arXiv  Detail & Related papers  (2025-04-10T20:19:50Z)
• Identifying and Mitigating API Misuse in Large Language Models [26.4403427473915]
  API misuse in code generated by large language models (LLMs) represents a serious emerging challenge in software development.<n>This paper presents the first comprehensive study of API misuse patterns in LLM-generated code, analyzing both method selection and parameter usage across Python and Java.<n>We propose Dr.Fix, a novel LLM-based automatic program repair approach for API misuse based on the aforementioned taxonomy.
  arXiv  Detail & Related papers  (2025-03-28T18:43:12Z)
• Reinforcement Learning for Long-Horizon Interactive LLM Agents [56.9860859585028]
  Interactive digital agents (IDAs) leverage APIs of stateful digital environments to perform tasks in response to user requests.<n>We present a reinforcement learning (RL) approach that trains IDAs directly in their target environments.<n>We derive LOOP, a data- and memory-efficient variant of proximal policy optimization.
  arXiv  Detail & Related papers  (2025-02-03T18:35:42Z)
• Utilizing API Response for Test Refinement [2.8002188463519944]
  This paper proposes a dynamic test refinement approach that leverages the response message.<n>Using an intelligent agent, the approach adds constraints to the API specification that are further used to generate a test scenario.<n>The proposed approach led to a decrease in the number of 4xx responses, taking a step closer to generating more realistic test cases.
  arXiv  Detail & Related papers  (2025-01-30T05:26:32Z)
• Your Fix Is My Exploit: Enabling Comprehensive DL Library API Fuzzing with Large Language Models [49.214291813478695]
  Deep learning (DL) libraries, widely used in AI applications, often contain vulnerabilities like overflows and use buffer-free errors. Traditional fuzzing struggles with the complexity and API diversity of DL libraries. We propose DFUZZ, an LLM-driven fuzzing approach for DL libraries.
  arXiv  Detail & Related papers  (2025-01-08T07:07:22Z)
• MMLU-CF: A Contamination-free Multi-task Language Understanding Benchmark [57.999567012489706]
  We propose a contamination-free and more challenging benchmark called MMLU-CF. This benchmark reassesses LLMs' understanding of world knowledge by averting both unintentional and malicious data leakage. Our evaluation of mainstream LLMs reveals that the powerful GPT-4o achieves merely a 5-shot score of 73.4% and a 0-shot score of 71.9% on the test set.
  arXiv  Detail & Related papers  (2024-12-19T18:58:04Z)
• ExploraCoder: Advancing code generation for multiple unseen APIs via planning and chained exploration [70.26807758443675]
  ExploraCoder is a training-free framework that empowers large language models to invoke unseen APIs in code solution.<n>We show that ExploraCoder significantly improves performance for models lacking prior API knowledge, achieving an absolute increase of 11.24% over niave RAG approaches and 14.07% over pretraining methods in pass@10.
  arXiv  Detail & Related papers  (2024-12-06T19:00:15Z)
• A Multi-Agent Approach for REST API Testing with Semantic Graphs and LLM-Driven Inputs [46.65963514391019]
  We present AutoRestTest, the first black-box tool to adopt a dependency-embedded multi-agent approach for REST API testing.<n>Our approach treats REST API testing as a separable problem, where four agents collaborate to optimize API exploration.<n>Our evaluation of AutoRestTest on 12 real-world REST services shows that it outperforms the four leading black-box REST API testing tools.
  arXiv  Detail & Related papers  (2024-11-11T16:20:27Z)
• Reinforcement Learning-Based REST API Testing with Multi-Coverage [4.127886193201882]
  MUCOREST is a novel Reinforcement Learning (RL)-based API testing approach that leverages Q-learning to maximize code coverage and output coverage. MUCOREST significantly outperforms state-of-the-art API testing approaches by 11.6-261.1% in the number of discovered API bugs.
  arXiv  Detail & Related papers  (2024-10-20T14:20:23Z)
• A Solution-based LLM API-using Methodology for Academic Information Seeking [49.096714812902576]
  SoAy is a solution-based LLM API-using methodology for academic information seeking. It uses code with a solution as the reasoning method, where a solution is a pre-constructed API calling sequence. Results show a 34.58-75.99% performance improvement compared to state-of-the-art LLM API-based baselines.
  arXiv  Detail & Related papers  (2024-05-24T02:44:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.