A Multi-Agent Approach for REST API Testing with Semantic Graphs and LLM-Driven Inputs

• URL: http://arxiv.org/abs/2411.07098v2
• Date: Wed, 22 Jan 2025 04:42:10 GMT
• Title: A Multi-Agent Approach for REST API Testing with Semantic Graphs and LLM-Driven Inputs
• Authors: Myeongsoo Kim, Tyler Stennett, Saurabh Sinha, Alessandro Orso,
• Abstract summary: We present AutoRestTest, the first black-box tool to adopt a dependency-embedded multi-agent approach for REST API testing. Our approach treats REST API testing as a separable problem, where four agents collaborate to optimize API exploration. Our evaluation of AutoRestTest on 12 real-world REST services shows that it outperforms the four leading black-box REST API testing tools.
• Score: 46.65963514391019
• License:
• Abstract: As modern web services increasingly rely on REST APIs, their thorough testing has become crucial. Furthermore, the advent of REST API documentation languages, such as the OpenAPI Specification, has led to the emergence of many black-box REST API testing tools. However, these tools often focus on individual test elements in isolation (e.g., APIs, parameters, values), resulting in lower coverage and less effectiveness in fault detection. To address these limitations, we present AutoRestTest, the first black-box tool to adopt a dependency-embedded multi-agent approach for REST API testing that integrates multi-agent reinforcement learning (MARL) with a semantic property dependency graph (SPDG) and Large Language Models (LLMs). Our approach treats REST API testing as a separable problem, where four agents -- API, dependency, parameter, and value agents -- collaborate to optimize API exploration. LLMs handle domain-specific value generation, the SPDG model simplifies the search space for dependencies using a similarity score between API operations, and MARL dynamically optimizes the agents' behavior. Our evaluation of AutoRestTest on 12 real-world REST services shows that it outperforms the four leading black-box REST API testing tools, including those assisted by RESTGPT (which generates realistic test inputs using LLMs), in terms of code coverage, operation coverage, and fault detection. Notably, AutoRestTest is the only tool able to trigger an internal server error in the Spotify service. Our ablation study illustrates that each component of AutoRestTest -- the SPDG, the LLM, and the agent-learning mechanism -- contributes to its overall effectiveness.

Related papers

• AutoRestTest: A Tool for Automated REST API Testing Using LLMs and MARL [46.65963514391019]
  AutoRestTest is a novel tool for testing REST APIs. It integrates the Semantic Operation Dependency Graph (SODG) with Multi-Agent Reinforcement Learning (MARL) and large language models (LLMs) It provides continuous telemetry on successful operation count, unique server errors detected, and time elapsed.
  arXiv  Detail & Related papers  (2025-01-15T05:54:33Z)
• LlamaRestTest: Effective REST API Testing with Small Language Models [50.058600784556816]
  We present LlamaRestTest, a novel approach that employs two custom LLMs to generate realistic test inputs. LlamaRestTest surpasses state-of-the-art tools in code coverage and error detection, even with RESTGPT-enhanced specifications.
  arXiv  Detail & Related papers  (2025-01-15T05:51:20Z)
• ExploraCoder: Advancing code generation for multiple unseen APIs via planning and chained exploration [70.26807758443675]
  ExploraCoder is a training-free framework that empowers large language models to invoke unseen APIs in code solution. We show that ExploraCoder significantly improves performance for models lacking prior API knowledge, achieving an absolute increase of 11.24% over niave RAG approaches and 14.07% over pretraining methods in pass@10.
  arXiv  Detail & Related papers  (2024-12-06T19:00:15Z)
• DeepREST: Automated Test Case Generation for REST APIs Exploiting Deep Reinforcement Learning [5.756036843502232]
  This paper introduces DeepREST, a novel black-box approach for automatically testing REST APIs. It leverages deep reinforcement learning to uncover implicit API constraints, that is, constraints hidden from API documentation. Our empirical validation suggests that the proposed approach is very effective in achieving high test coverage and fault detection.
  arXiv  Detail & Related papers  (2024-08-16T08:03:55Z)
• KAT: Dependency-aware Automated API Testing with Large Language Models [1.7264233311359707]
  KAT (Katalon API Testing) is a novel AI-driven approach that autonomously generates test cases to validate APIs. Our evaluation of KAT using 12 real-world services shows that it can improve validation coverage, detect more undocumented status codes, and reduce false positives in these services.
  arXiv  Detail & Related papers  (2024-07-14T14:48:18Z)
• You Can REST Now: Automated Specification Inference and Black-Box Testing of RESTful APIs with Large Language Models [8.753312212588371]
  manually documenting APIs is a time-consuming and error-prone task, resulting in unavailable, incomplete, or imprecise documentation. Recently, Large Language Models (LLMs) have demonstrated exceptional abilities to automate tasks based on their colossal training data. We present RESTSpecIT, the first automated API specification inference and black-box testing approach.
  arXiv  Detail & Related papers  (2024-02-07T18:55:41Z)
• Leveraging Large Language Models to Improve REST API Testing [51.284096009803406]
  RESTGPT takes as input an API specification, extracts machine-interpretable rules, and generates example parameter values from natural-language descriptions in the specification. Our evaluations indicate that RESTGPT outperforms existing techniques in both rule extraction and value generation.
  arXiv  Detail & Related papers  (2023-12-01T19:53:23Z)
• Adaptive REST API Testing with Reinforcement Learning [54.68542517176757]
  Current testing tools lack efficient exploration mechanisms, treating all operations and parameters equally. Current tools struggle when response schemas are absent in the specification or exhibit variants. We present an adaptive REST API testing technique incorporates reinforcement learning to prioritize operations during exploration.
  arXiv  Detail & Related papers  (2023-09-08T20:27:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.