RESTOR: Automated Test Oracle Generation for RESTful APIs via Reinforcement Learning

• URL: http://arxiv.org/abs/2607.23963v1
• Date: Mon, 27 Jul 2026 03:20:28 GMT
• Title: RESTOR: Automated Test Oracle Generation for RESTful APIs via Reinforcement Learning
• Abstract summary: Restor is a framework that generates executable test assertions from a single observed request-response pair in a black-box setting.<n>We evaluate Restor on an industrial dataset comprising over 2,300 API traces across 246 real-world services.
• Score: 21.927343858709122
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Modern REST API testing faces a critical challenge in defining reliable test oracles, particularly in agile industrial environments where formal specifications (e.g., OpenAPI) are frequently missing or outdated, and historical execution logs are unavailable for newly deployed endpoints. In this paper, we present Restor (Reinforcement Enhanced Single-Traffic Oracle generator for REST APIs), a framework that generates executable test assertions from a single observed request-response pair in a black-box setting. Unlike existing approaches that rely on rule-based templates or massive training logs, Restor utilizes a novel data augmentation pipeline to fine-tune a lightweight Large Language Model (LLM) via Group Relative Policy Optimization (GRPO). This training process enables the model to internalize testing "common sense" by optimizing a reward function that jointly encourages: (i) the selection of stable, semantically meaningful fields for validation and the avoidance of dynamic noise (e.g., timestamps or trace IDs); (ii) the generation of robust assertions that withstand logic variations. We evaluate Restor on an industrial dataset comprising over 2,300 API traces across 246 real-world services. Comprehensive experiments demonstrate that Restor significantly outperforms prompt-engineered baselines and generalist models, achieving a superior $F_1$ score of 85.42% in key field identification and increasing the proportion of semantically accurate assertions. Furthermore, deployment in a production CI/CD workflow at ByteDance confirms its practical value: the system raised the adoption rate of automatically generated test cases from 74.1% to over 96%, substantially reducing manual Quality Assurance (QA) effort while ensuring high execution stability.

Related papers

• Industrial Practice of LLM-Based Test Case Carving and Assertion Generation (Experience Paper) [14.766370287473519]
  This paper presents NL2Test, an end-to-end approach and tool that generates executable API regression tests from natural-language scenarios.<n> NL2Test addresses two coupled tasks: test case carving and assertion generation.<n>We evaluate NL2Test on 51 industrial regression scenarios extracted from a large consumer-facing Internet company.
  arXiv  Detail & Related papers  (2026-07-27T04:52:51Z)
• ComplexMCP: Evaluation of LLM Agents in Dynamic, Interdependent, and Large-Scale Tool Sandbox [61.862814740220806]
  $textbfComplexMCP$ is a benchmark designed to evaluate agents in rigorous conditions.<n>Built on the Model Context Protocol (MCP), $textbfComplexMCP$ provides over 300 meticulously tested tools.
  arXiv  Detail & Related papers  (2026-05-11T16:20:51Z)
• LiveFMBench: Unveiling the Power and Limits of Agentic Workflows in Specification Generation [75.05397479715576]
  Large language models (LLMs) and agents have shown promising progress, but their true capabilities and failure modes remain unclear.<n>We present the first systematic and contamination-aware study of LLM- and agent-based formal specification generation for C programs.
  arXiv  Detail & Related papers  (2026-05-02T11:31:33Z)
• Rewarding the Scientific Process: Process-Level Reward Modeling for Agentic Data Analysis [68.28714988482703]
  Process Reward Models (PRMs) have achieved remarkable success in augmenting the reasoning capabilities of Large Language Models (LLMs)<n>We present a study revealing that general-domain PRMs struggle to supervise data analysis agents.<n>We introduce DataPRM, a novel environment-aware generative process reward model.
  arXiv  Detail & Related papers  (2026-04-27T09:00:30Z)
• Compiled AI: Deterministic Code Generation for LLM-Based Workflow Automation [3.225273674498579]
  We study compiled AI, a paradigm in which large language models generate executable code artifacts during a compilation phase, after which execute deterministically without further model invocation.<n>Our contribution is a systems-oriented study of its application to high-stakes runtime enterprise.
  arXiv  Detail & Related papers  (2026-04-06T20:25:20Z)
• ProdCodeBench: A Production-Derived Benchmark for Evaluating AI Coding Agents [10.578603956693696]
  This paper presents a methodology for curating production-derived benchmarks, illustrated through ProdCodeBench.<n>Each curated sample consists of a verbatim prompt, a committed code change and fail-to-pass tests spanning seven programming languages.<n>Our systematic analysis of four foundation models yields solve rates ranging from 53.2% to 72.2%.
  arXiv  Detail & Related papers  (2026-04-02T01:52:55Z)
• Dynamic analysis enhances issue resolution [53.50448142467294]
  DAIRA (Dynamic Analysis-enhanced Issue Resolution Agent) is an automated repair framework that embeds dynamic analysis into the agent's reasoning cycle.<n> Driven by a Test Tracing-Driven methodology, DAIRA utilizes lightweight monitors to extract critical runtime data.<n>When powered by Gemini 3 Flash Preview, DAIRA establishes a new state-of-the-art (SOTA) performance, achieving a 79.4% resolution rate on the SWE-bench Verified dataset.
  arXiv  Detail & Related papers  (2026-03-23T14:48:54Z)
• BOSQTGEN: Breaking the Sound Barrier in Test Generation [3.052470294814771]
  We introduce BOSQTGEN, a novel black-box and tool for API test generation.<n> BOSQTGEN utilizes a novel approach for decomposing API specifications into primitives, using LLMs to suggest coherent interactions for them, and employing testing to efficiently sample over these values.<n>The resulting BOSQTGEN system achieves an average of 82% of critical code coverage on benchmarks, often a 20% or more increase over prior state-of-the-art systems.
  arXiv  Detail & Related papers  (2025-10-22T17:11:30Z)
• LLMShot: Reducing snapshot testing maintenance via LLMs [0.5218155982819203]
  Snapshot testing has emerged as a critical technique for UI validation in modern software development.<n>This paper introduces LLMShot, a novel framework that leverages Vision-Language Models (VLMs) to automatically analyze snapshot test failures.
  arXiv  Detail & Related papers  (2025-07-14T08:47:19Z)
• Are You Getting What You Pay For? Auditing Model Substitution in LLM APIs [71.7892165868749]
  Commercial Large Language Model (LLM) APIs create a fundamental trust problem.<n>Users pay for specific models but have no guarantee that providers deliver them faithfully.<n>We formalize this model substitution problem and evaluate detection methods under realistic adversarial conditions.<n>We propose and evaluate the use of Trusted Execution Environments (TEEs) as one practical and robust solution.
  arXiv  Detail & Related papers  (2025-04-07T03:57:41Z)
• Thinking Longer, Not Larger: Enhancing Software Engineering Agents via Scaling Test-Time Compute [61.00662702026523]
  We propose a unified Test-Time Compute scaling framework that leverages increased inference-time instead of larger models.<n>Our framework incorporates two complementary strategies: internal TTC and external TTC.<n>We demonstrate our textbf32B model achieves a 46% issue resolution rate, surpassing significantly larger models such as DeepSeek R1 671B and OpenAI o1.
  arXiv  Detail & Related papers  (2025-03-31T07:31:32Z)
• AutoPT: How Far Are We from the End2End Automated Web Penetration Testing? [54.65079443902714]
  We introduce AutoPT, an automated penetration testing agent based on the principle of PSM driven by LLMs. Our results show that AutoPT outperforms the baseline framework ReAct on the GPT-4o mini model.
  arXiv  Detail & Related papers  (2024-11-02T13:24:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.