Related papers: MASTOR: A Multi-Agent Approach to Semantic Test Oracle Generation for RESTful APIs

MASTOR: A Multi-Agent Approach to Semantic Test Oracle Generation for RESTful APIs

URL: http://arxiv.org/abs/2606.10465v1
Date: Tue, 09 Jun 2026 06:35:21 GMT
Title: MASTOR: A Multi-Agent Approach to Semantic Test Oracle Generation for RESTful APIs
Abstract summary: Existing automated API testing approaches rely on simple checks.<n>We propose MASTOR, a Multi-Agent approach for generating Semantic Test Oracles for APIs based on implementation source code.
Score: 8.248920932579876
License: http://creativecommons.org/licenses/by/4.0/
Abstract: Existing automated RESTful API testing approaches commonly rely on simple checks (e.g., HTTP status codes, schema conformance), which are insufficient for detecting semantic faults, business logic violations, and state-dependent inconsistencies. To address this, we propose MASTOR, a Multi-Agent approach for generating Semantic Test Oracles for RESTful APIs based on implementation source code. MASTOR consists of two phases: source analysis and oracle generation. The former employs a source extraction agent to construct a source context for each endpoint operation by analyzing a transitive import closure of relevant source files. The latter employs two parallel oracle-generation paths over the collected contexts: a single-operation path producing status and field oracles per operation, and a multi-operation path generating behavioral consistency oracles for operation sequences by leveraging cross-operation semantic associations. Both paths apply a challenger-agent review, where a dedicated reviewer identifies weaknesses and issues improvement hints to guide targeted regeneration, followed by oracle normalization to filter out structurally invalid oracles. We evaluated MASTOR on a benchmark of 13 open-source RESTful API projects (296 operations, 251,303 lines of code) from the WFD and PRAB datasets. MASTOR achieved an average mutation score of 75.4%, generating 10,022 oracles. These oracles were translated into executable assertions via ToJUnit and ToPostmanAssertify, and into human-readable descriptions via ToReadable. In a baseline comparison on 50 selected operations, MASTOR outperformed Direct Prompting by 30.1 percentage points (69.9% vs. 39.8%) and SATORI by 49.4 percentage points (69.9% vs. 20.5%).

Related papers

Identifying Latent Declarative Representations of Code for Assisting Repository Migration [0.7763235307178853]
ADFD-Migrate approximates the latent representation with an annotated data-flow diagram (ADFD) of processes, data stores, external entities, flows, and behavioral contracts.<n>We evaluate ADFD-Migrate on f2x50, a new benchmark of 50Fortran repositories spanning 1.5k--1.6M lines of code and three complexity tiers.
arXiv Detail & Related papers (2026-08-22T08:23:14Z)
A-SR: Self-Evolving Agentic LLMs for Symbolic Regression via Hierarchical Coordination [51.06539604709775]
We propose A-SR, a self-evolving agentic framework that shifts the control unit from expression edits to role-conditioned evidence views.<n>A-SR coordinates formula discovery through routing among coordination protocols, an online evaluator-reward role policy, and state-routed process memory.
arXiv Detail & Related papers (2026-08-05T14:01:10Z)
LogicHunter: Testing LLM Agent Frameworks with an Agentic Oracle [6.653693552089508]
Large Language Model (LLM) agent frameworks such as LangChain, LlamaIndex, and CrewAI have become critical infrastructure powering production AI systems.<n> defects in these pure Python frameworks manifest as ordinary exceptions or silent semantic failures, creating profound oracle ambiguity.<n>We present LogicHunter, a fuzzing framework that addresses both the generation and oracle challenges through active specification-aware testing.
arXiv Detail & Related papers (2026-07-07T12:21:42Z)
BADGER: Bridging Agentic and Deterministic Evaluation for Generative Enterprise Reasoning [0.8681424810073922]
BADGER is a unified framework integrating text-to-assessment with agentic behavior evaluation.<n> BADGER runs entirely within client's governed data environment.<n>It supports rapid prototyping of client-specific judges and metrics.
arXiv Detail & Related papers (2026-06-01T11:42:37Z)
iCoRe: An Iterative Correlation-Aware Retriever for Bug Reproduction Test Generation [10.824024745675281]
iCoRe is an iterative, correlation-aware context retrieval approach.<n>Our method achieves a Fail-to-Pass rate of 42.0% and 52.8% respectively.
arXiv Detail & Related papers (2026-04-21T08:26:30Z)
APEX-EM: Non-Parametric Online Learning for Autonomous Agents via Structured Procedural-Episodic Experience Replay [7.370176470430802]
LLM-based autonomous agents lack persistent procedural memory.<n>We present APEX-EM, a non-parametric online learning framework that accumulates, retrieves, and reuses structured procedural plans.
arXiv Detail & Related papers (2026-03-31T00:24:56Z)
Beyond Relevance: On the Relationship Between Retrieval and RAG Information Coverage [89.58253972744531]
Retrieval-augmented generation (RAG) systems combine document retrieval with a generative model to address complex information seeking tasks like report generation.<n>We investigate whether upstream retrieval metrics can serve as reliable early indicators of the final generated response's information coverage.<n>Our findings demonstrate strong correlations between coverage-based retrieval metrics and nugget coverage in generated responses at both topic and system levels.
arXiv Detail & Related papers (2026-03-09T18:20:20Z)
CaveAgent: Transforming LLMs into Stateful Runtime Operators [31.548422546991915]
We present CaveAgent, a framework that transforms the paradigm from "LLM-as-Text-Generator" to "LLM-as-As-Runtime-Runtime"<n>CaveAgent achieves a 10.5% success rate improvement on retail tasks and reduces total token consumption by 28.4% in multi-turn scenarios.
arXiv Detail & Related papers (2026-01-04T15:32:47Z)
Eigen-1: Adaptive Multi-Agent Refinement with Monitor-Based RAG for Scientific Reasoning [53.45095336430027]
We develop a unified framework that combines implicit retrieval and structured collaboration.<n>On Humanity's Last Exam (HLE) Bio/Chem Gold, our framework achieves 48.3% accuracy.<n>Results on SuperGPQA and TRQA confirm robustness across domains.
arXiv Detail & Related papers (2025-09-25T14:05:55Z)
Benchmarking Deep Search over Heterogeneous Enterprise Data [73.55304268238474]
We present a new benchmark for evaluating a form of retrieval-augmented generation (RAG)<n>RAG requires source-aware, multi-hop reasoning over diverse, sparsed, but related sources.<n>We build it using a synthetic data pipeline that simulates business across product planning, development, and support stages.
arXiv Detail & Related papers (2025-06-29T08:34:59Z)
Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systems [50.29939179830491]
Failure attribution in LLM multi-agent systems remains underexplored and labor-intensive.<n>We develop and evaluate three automated failure attribution methods, summarizing their corresponding pros and cons.<n>The best method achieves 53.5% accuracy in identifying failure-responsible agents but only 14.2% in pinpointing failure steps.
arXiv Detail & Related papers (2025-04-30T23:09:44Z)
SOPBench: Evaluating Language Agents at Following Standard Operating Procedures and Constraints [59.645885492637845]
SOPBench is an evaluation pipeline that transforms each service-specific SOP code program into a directed graph of executable functions.<n>Our approach transforms each service-specific SOP code program into a directed graph of executable functions and requires agents to call these functions based on natural language SOP descriptions.<n>We evaluate 18 leading models, and results show the task is challenging even for top-tier models.
arXiv Detail & Related papers (2025-03-11T17:53:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.