You Can REST Now: Automated REST API Documentation and Testing via LLM-Assisted Request Mutations

• URL: http://arxiv.org/abs/2402.05102v2
• Date: Tue, 15 Jul 2025 19:25:43 GMT
• Title: You Can REST Now: Automated REST API Documentation and Testing via LLM-Assisted Request Mutations
• Authors: Alix Decrop, Xavier Devroey, Mike Papadakis, Pierre-Yves Schobbens, Gilles Perrouin,
• Abstract summary: We present RESTSpecIT, the first automated approach that infers documentation and performs black-box testing of REST APIs.<n>Our approach requires minimal user input compared to state-of-the-art tools.<n>We evaluate the quality of our tool with three state-of-the-art LLMs: DeepSeek V3, GPT-4.1, and GPT-3.5.
• Score: 8.158964648211002
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: REST APIs are prevalent among web service implementations, easing interoperability through the HTTP protocol. API testers and users exploit the widely adopted OpenAPI Specification (OAS), a machine-readable standard to document REST APIs. However, documenting APIs is a time-consuming and error-prone task, and existing documentation is not always complete, publicly accessible, or up-to-date. This situation limits the efficiency of testing tools and hinders human comprehension. Large Language Models (LLMs) offer the potential to automatically infer API documentation, using their colossal training data. In this paper, we present RESTSpecIT, the first automated approach that infers documentation and performs black-box testing of REST APIs by leveraging LLMs. Our approach requires minimal user input compared to state-of-the-art tools; Given an API name and an LLM access key, RESTSpecIT generates API request seeds and mutates them with data returned by the LLM. The tool then analyzes API responses for documentation inference and testing purposes. RESTSpecIT utilizes an in-context prompt masking strategy, requiring no prior model fine-tuning. We evaluate the quality of our tool with three state-of-the-art LLMs: DeepSeek V3, GPT-4.1, and GPT-3.5. Our evaluation demonstrates that RESTSpecIT can (1) infer documentation with 88.62% of routes and 89.25% of query parameters found on average, (2) discover undocumented API data, (3) operate efficiently (in terms of model costs, requests sent, runtime), and (4) assist REST API testing by uncovering server errors and generating valid OpenAPI Specification inputs for testing tools.

Related papers

• LRASGen: LLM-based RESTful API Specification Generation [3.420331911153286]
  We propose a novel approach for generating the OpenAPI Specification (OAS) specifications for APIs using Large Language Models (LLMs) Compared with existing tools and methods, LRASGen can generate the OASs, even when the implementation is incomplete (with partial code, annotations/comments, etc.) LRASGen-generated specifications cover an average of 48.85% more missed entities than the developer-provided specifications.
  arXiv  Detail & Related papers  (2025-04-23T15:52:50Z)
• Test Amplification for REST APIs via Single and Multi-Agent LLM Systems [1.6499388997661122]
  We show how single-agent and multi-agent LLM systems can amplify a REST API test suite. Our evaluation demonstrates increased API coverage, identification of numerous bugs in the API under test, and insights into the computational cost and energy consumption of both approaches.
  arXiv  Detail & Related papers  (2025-04-10T20:19:50Z)
• AutoRestTest: A Tool for Automated REST API Testing Using LLMs and MARL [46.65963514391019]
  AutoRestTest is a novel tool that integrates the Semantic Property Dependency Graph (SPDG) with Multi-Agent Reinforcement Learning (MARL) and large language models (LLMs) for effective REST API testing.
  arXiv  Detail & Related papers  (2025-01-15T05:54:33Z)
• LlamaRestTest: Effective REST API Testing with Small Language Models [50.058600784556816]
  We present LlamaRestTest, a novel approach that employs two custom Large Language Models (LLMs) to generate realistic test inputs. We evaluate it against several state-of-the-art REST API testing tools, including RESTGPT, a GPT-powered specification-enhancement tool. Our study shows that small language models can perform as well as, or better than, large language models in REST API testing.
  arXiv  Detail & Related papers  (2025-01-15T05:51:20Z)
• APIRL: Deep Reinforcement Learning for REST API Fuzzing [3.053989095162017]
  APIRL is a fully automated deep reinforcement learning tool for testing REST APIs. We show APIRL can find significantly more bugs than the state-of-the-art in real world REST APIs.
  arXiv  Detail & Related papers  (2024-12-20T15:40:51Z)
• A Multi-Agent Approach for REST API Testing with Semantic Graphs and LLM-Driven Inputs [46.65963514391019]
  We present AutoRestTest, the first black-box framework to adopt a dependency-embedded multi-agent approach for REST API testing. We integrate Multi-Agent Reinforcement Learning (MARL) with a Semantic Property Dependency Graph (SPDG) and Large Language Models (LLMs) Our approach treats REST API testing as a separable problem, where four agents -- API, dependency, parameter, and value -- collaborate to optimize API exploration.
  arXiv  Detail & Related papers  (2024-11-11T16:20:27Z)
• A Systematic Evaluation of Large Code Models in API Suggestion: When, Which, and How [53.65636914757381]
  API suggestion is a critical task in modern software development. Recent advancements in large code models (LCMs) have shown promise in the API suggestion task.
  arXiv  Detail & Related papers  (2024-09-20T03:12:35Z)
• DeepREST: Automated Test Case Generation for REST APIs Exploiting Deep Reinforcement Learning [5.756036843502232]
  This paper introduces DeepREST, a novel black-box approach for automatically testing REST APIs. It leverages deep reinforcement learning to uncover implicit API constraints, that is, constraints hidden from API documentation. Our empirical validation suggests that the proposed approach is very effective in achieving high test coverage and fault detection.
  arXiv  Detail & Related papers  (2024-08-16T08:03:55Z)
• KAT: Dependency-aware Automated API Testing with Large Language Models [1.7264233311359707]
  KAT (Katalon API Testing) is a novel AI-driven approach that autonomously generates test cases to validate APIs. Our evaluation of KAT using 12 real-world services shows that it can improve validation coverage, detect more undocumented status codes, and reduce false positives in these services.
  arXiv  Detail & Related papers  (2024-07-14T14:48:18Z)
• A Solution-based LLM API-using Methodology for Academic Information Seeking [49.096714812902576]
  SoAy is a solution-based LLM API-using methodology for academic information seeking. It uses code with a solution as the reasoning method, where a solution is a pre-constructed API calling sequence. Results show a 34.58-75.99% performance improvement compared to state-of-the-art LLM API-based baselines.
  arXiv  Detail & Related papers  (2024-05-24T02:44:14Z)
• Leveraging Large Language Models to Improve REST API Testing [51.284096009803406]
  RESTGPT takes as input an API specification, extracts machine-interpretable rules, and generates example parameter values from natural-language descriptions in the specification. Our evaluations indicate that RESTGPT outperforms existing techniques in both rule extraction and value generation.
  arXiv  Detail & Related papers  (2023-12-01T19:53:23Z)
• Exploring Behaviours of RESTful APIs in an Industrial Setting [0.43012765978447565]
  We propose a set of behavioural properties, common to REST APIs, which are used to generate examples of behaviours that these APIs exhibit. These examples can be used both (i) to further the understanding of the API and (ii) as a source of automatic test cases. Our approach can generate examples deemed relevant for understanding the system and for a source of test generation by practitioners.
  arXiv  Detail & Related papers  (2023-10-26T11:33:11Z)
• Adaptive REST API Testing with Reinforcement Learning [54.68542517176757]
  Current testing tools lack efficient exploration mechanisms, treating all operations and parameters equally. Current tools struggle when response schemas are absent in the specification or exhibit variants. We present an adaptive REST API testing technique incorporates reinforcement learning to prioritize operations during exploration.
  arXiv  Detail & Related papers  (2023-09-08T20:27:05Z)
• RestGPT: Connecting Large Language Models with Real-World RESTful APIs [44.94234920380684]
  A tool-augmented large language models (LLMs) have achieved remarkable progress in tackling a broad range of tasks. To address the practical challenges of tackling complex instructions, we propose RestGPT, which exploits the power of robustness. To fully evaluate RestGPT, we propose RestBench, a high-quality benchmark which consists of two real-world scenarios and human-annotated instructions.
  arXiv  Detail & Related papers  (2023-06-11T08:53:12Z)
• Carving UI Tests to Generate API Tests and API Specification [8.743426215048451]
  API-level testing can play an important role, in-between unit-level testing and UI-level (or end-to-end) testing. Existing API testing tools require API specifications, which often may not be available or, when available, be inconsistent with the API implementation. We present an approach that leverages UI testing to enable API-level testing for web applications.
  arXiv  Detail & Related papers  (2023-05-24T03:53:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.