Sakura: An Approach for Generating Complex Tests from Natural Language Test Descriptions

• URL: http://arxiv.org/abs/2606.00530v1
• Date: Sat, 30 May 2026 04:49:10 GMT
• Title: Sakura: An Approach for Generating Complex Tests from Natural Language Test Descriptions
• Abstract summary: Sakura is a framework for generating tests from natural language (NL) descriptions of developer intent.<n>Sakura decomposes NL descriptions into structured blocks and processes them using a multi-agent system.<n>Sakura significantly outperforms off-the-truth agentic tools such as Gemini CLI.
• Score: 46.562528718959506
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Testing is a core activity in software development workflows, and research on its automation has spanned several decades. Most existing approaches generate unit tests for individual methods, validate isolated API endpoints, or target user interface (UI) layers, with non-API and non-UI automated test generators typically exercising only a single focal method. Recent empirical evidence shows a substantial gap between such generated tests and developer-written ones, which often span multiple focal methods, involve complex call sequences, and contain elaborate assertions that current automated approaches fail to capture. To address this gap, we propose generating tests from natural language (NL) descriptions of developer intent. We present Sakura, the first agent-based framework for generating structurally complex test cases from NL descriptions. Sakura decomposes NL descriptions into structured blocks and processes them using a multi-agent system consisting of a localization agent that grounds test steps in concrete application code via static analysis, a composition agent that synthesizes compilable test code and iteratively refines it using execution feedback, and a supervisor agent that coordinates agent interactions. To evaluate Sakura, we curate a novel dataset of NL test descriptions at three levels of abstraction, systematically generated from developer-written tests mined from Apache Commons projects. Across 20 applications and 1,464 test scenarios, Sakura significantly outperforms off-the-shelf agentic tools such as Gemini CLI. Specifically, Sakura achieves 50-78% higher test compilability and 38-66% higher coverage overlap with ground-truth tests compared to baselines using the same models. Moreover, Sakura paired with small open-source models such as Devstral Small 2 and Qwen3-Coder outperforms Gemini CLI using large proprietary models, while also being more cost-effective.

Related papers

• Multi-Agent LLM Collaboration for Unit Test Generation via Human-Testing-Inspired Workflows [18.32804849870551]
  Large Language Models (LLMs) have spurred a surge of research into automated unit test generation.<n>We propose TestAgent, an LLM-based test generation approach that emulates human testing practices via a multi-agent collaboration mechanism.<n>TestAgent achieves 97.46% execution rate, 92.34% line coverage, 90.24% branch coverage, and 83.69% mutation score on six Java projects.
  arXiv  Detail & Related papers  (2026-07-10T05:16:54Z)
• Testing with AI Agents: An Empirical Study of Test Generation Frequency, Quality, and Coverage [1.6782288954198918]
  This study presents an empirical analysis of test generation by agent-based coding tools using the AIDev dataset.<n>We extracted 2,232 commits containing test-related changes and investigated three aspects: the frequency of test additions, the structural characteristics of the generated tests, and their impact on code coverage.<n>Our findings reveal that (i) AI authored 16.4% of all commits adding tests in real-world repositories, (ii) AI-generated test methods exhibit distinct structural patterns, featuring longer code and a higher density of assertions, and (iii) AI-generated tests contribute to code coverage comparable to human-written tests
  arXiv  Detail & Related papers  (2026-03-14T03:01:44Z)
• SAINT: Service-level Integration Test Generation with Program Analysis and LLM-based Agents [43.3273990835497]
  SAINT is a novel white-box testing approach for service-level testing of enterprise Java applications.<n> SAINT combines static analysis, large language models (LLMs), and LLM-based agents to automatically generate endpoint and scenario-based tests.
  arXiv  Detail & Related papers  (2025-11-17T12:29:42Z)
• Rethinking Testing for LLM Applications: Characteristics, Challenges, and a Lightweight Interaction Protocol [83.83217247686402]
  Large Language Models (LLMs) have evolved from simple text generators into complex software systems that integrate retrieval augmentation, tool invocation, and multi-turn interactions.<n>Their inherent non-determinism, dynamism, and context dependence pose fundamental challenges for quality assurance.<n>This paper decomposes LLM applications into a three-layer architecture: textbftextitSystem Shell Layer, textbftextitPrompt Orchestration Layer, and textbftextitLLM Inference Core.
  arXiv  Detail & Related papers  (2025-08-28T13:00:28Z)
• Alignment with Fill-In-the-Middle for Enhancing Code Generation [56.791415642365415]
  We propose a novel approach that splits code snippets into smaller, granular blocks, creating more diverse DPO pairs from the same test cases.<n>Our approach demonstrates significant improvements in code generation tasks, as validated by experiments on benchmark datasets such as HumanEval (+), MBPP (+), APPS, LiveCodeBench, and BigCodeBench.
  arXiv  Detail & Related papers  (2025-08-27T03:15:53Z)
• LlamaRestTest: Effective REST API Testing with Small Language Models [50.058600784556816]
  We present LlamaRestTest, a novel approach that employs two custom Large Language Models (LLMs) to generate realistic test inputs.<n>We evaluate it against several state-of-the-art REST API testing tools, including RESTGPT, a GPT-powered specification-enhancement tool.<n>Our study shows that small language models can perform as well as, or better than, large language models in REST API testing.
  arXiv  Detail & Related papers  (2025-01-15T05:51:20Z)
• Commit0: Library Generation from Scratch [77.38414688148006]
  Commit0 is a benchmark that challenges AI agents to write libraries from scratch.<n>Agents are provided with a specification document outlining the library's API as well as a suite of interactive unit tests.<n> Commit0 also offers an interactive environment where models receive static analysis and execution feedback on the code they generate.
  arXiv  Detail & Related papers  (2024-12-02T18:11:30Z)
• ASTER: Natural and Multi-language Unit Test Generation with LLMs [6.259245181881262]
  We describe a generic pipeline that incorporates static analysis to guide LLMs in generating compilable and high-coverage test cases.<n>We conduct an empirical study to assess the quality of the generated tests in terms of code coverage and test naturalness.
  arXiv  Detail & Related papers  (2024-09-04T21:46:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.