TestMap: Evidence Infrastructure for Foundation-Model-Assisted Test Generation

• URL: http://arxiv.org/abs/2606.10211v1
• Date: Mon, 08 Jun 2026 22:04:51 GMT
• Title: TestMap: Evidence Infrastructure for Foundation-Model-Assisted Test Generation
• Abstract summary: This paper presents TestMap, an open-source prototype that automates foundation-model-assisted test generation for C#/.NET repositories.<n>TestMap supports repository analysis, source-test mapping, baseline execution, code metric collection, test smell detection, coverage measurement, mutation testing, model-guided test generation, validation, repair, and repository-specific experiment tracking.
• Score: 1.1546538067103256
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Foundation models (FMs) can generate plausible unit tests, but determining whether those tests are correct, useful, maintainable, and worth integrating remains difficult. Generated tests must be mapped to the code they target, inserted into real projects, built, executed, measured against the baseline suite, repaired when necessary, and compared across models and generation strategies. This validation process is fragmented across build systems, test runners, coverage tools, mutation tools, static analyzers, and experiment scripts. The problem is especially important because generated tests are both code artifacts and validation artifacts: they must themselves be validated before they can be trusted as evidence about the system under test. This paper presents TestMap, an open-source infrastructure prototype that automates evidence-backed foundation-model-assisted test generation for C#/.NET repositories. TestMap supports repository analysis, source-test mapping, baseline execution, code metric collection, test smell detection, coverage measurement, mutation testing, model-guided test generation, validation, repair, and repository-specific experiment tracking. Rather than reporting only final passing tests, TestMap records the lifecycle of each generated candidate, including failed, repaired, low-impact, and evidence positive outcomes. These intermediate outcomes can reveal model limitations, missing context, repair cost, toolchain inefficiencies, or possible faults in the system under test. Using TestMap as a design case, we describe the architecture and evidence model needed to make generated tests observable, repeatable, and comparable across repositories, models, prompts, and generation strategies. We conclude with lessons learned and open challenges, including oracle and assertion quality, metric attribution, test maintainability, flakiness, execution cost, and developer acceptance.

Related papers

• Test-Driven Approaches to Software Engineering with Large Language Models: A Survey of Phases, Tasks, and Agent Skills [14.44743150931186]
  Tests increasingly participate in the decisions made by large language models and software engineering agents.<n>They specify intended behavior, guide program construction and repair, select candidates, constrain transformations, and provide execution evidence for software analysis.<n>These uses draw on test-driven development, yet differ substantially in test order, oracle availability, editable artifacts, and the role of execution.
  arXiv  Detail & Related papers  (2026-09-10T06:20:22Z)
• Improving LLM-Driven Test Generation by Learning from Mocking Information [3.7334264501860375]
  MOCKMILL generates test cases by exploiting mocking information automatically extracted from developer-written tests.<n>We evaluated MOCKMILL on 10 open-source classes from six Java projects using four Large Language Models.
  arXiv  Detail & Related papers  (2026-04-21T10:24:26Z)
• WebTestBench: Evaluating Computer-Use Agents towards End-to-End Automated Web Testing [57.7131457251794]
  We introduce WebTestBench, a benchmark for evaluating end-to-end automated web testing.<n>We decompose the testing process into two cascaded sub-tasks, checklist generation and defect detection, and propose WebTester.<n>Our findings expose a substantial gap between current computer-use agent capabilities and industrial-grade deployment demands.
  arXiv  Detail & Related papers  (2026-03-26T09:27:29Z)
• Code-A1: Adversarial Evolving of Code LLM and Test LLM via Reinforcement Learning [54.95476453942411]
  Reinforcement learning for code generation relies on verifiable rewards from unit test pass rates.<n>Recent self-play methods unify code and test generation in a single model.<n>Code-A1 achieves code generation performance matching or exceeding models trained on human-annotated tests.
  arXiv  Detail & Related papers  (2026-03-16T17:58:13Z)
• LLMs for Automated Unit Test Generation and Assessment in Java: The AgoneTest Framework [2.501198441875755]
  AgoneTest is an evaluation framework for Large Language Model-generated unit tests in Java.<n>For the subset of tests that compile, LLM-generated tests can match or exceed human-written tests in terms of coverage and defect detection.
  arXiv  Detail & Related papers  (2025-11-25T15:33:00Z)
• Intention-Driven Generation of Project-Specific Test Cases [45.2380093475221]
  We propose IntentionTest, which generates project-specific tests given the description of validation intention.<n>We extensively evaluate IntentionTest against state-of-the-art baselines (DA, ChatTester, and EvoSuite) on 4,146 test cases from 13 open-source projects.
  arXiv  Detail & Related papers  (2025-07-28T08:35:04Z)
• LlamaRestTest: Effective REST API Testing with Small Language Models [50.058600784556816]
  We present LlamaRestTest, a novel approach that employs two custom Large Language Models (LLMs) to generate realistic test inputs.<n>We evaluate it against several state-of-the-art REST API testing tools, including RESTGPT, a GPT-powered specification-enhancement tool.<n>Our study shows that small language models can perform as well as, or better than, large language models in REST API testing.
  arXiv  Detail & Related papers  (2025-01-15T05:51:20Z)
• Context-Aware Testing: A New Paradigm for Model Testing with Large Language Models [49.06068319380296]
  We introduce context-aware testing (CAT) which uses context as an inductive bias to guide the search for meaningful model failures. We instantiate the first CAT system, SMART Testing, which employs large language models to hypothesize relevant and likely failures.
  arXiv  Detail & Related papers  (2024-10-31T15:06:16Z)
• TestGenEval: A Real World Unit Test Generation and Test Completion Benchmark [24.14654309612826]
  TestGenEval comprises 68,647 tests from 1,210 code and test file pairs across 11 well-maintained Python repositories.<n>It covers initial tests authoring, test suite completion, and code coverage improvements.<n>We evaluate several popular models, with sizes ranging from 7B to 405B parameters.
  arXiv  Detail & Related papers  (2024-10-01T14:47:05Z)
• GPT-HateCheck: Can LLMs Write Better Functional Tests for Hate Speech Detection? [50.53312866647302]
  HateCheck is a suite for testing fine-grained model functionalities on synthesized data. We propose GPT-HateCheck, a framework to generate more diverse and realistic functional tests from scratch. Crowd-sourced annotation demonstrates that the generated test cases are of high quality.
  arXiv  Detail & Related papers  (2024-02-23T10:02:01Z)
• Automatic Generation of Test Cases based on Bug Reports: a Feasibility Study with Large Language Models [4.318319522015101]
  Existing approaches produce test cases that either can be qualified as simple (e.g. unit tests) or that require precise specifications. Most testing procedures still rely on test cases written by humans to form test suites. We investigate the feasibility of performing this generation by leveraging large language models (LLMs) and using bug reports as inputs.
  arXiv  Detail & Related papers  (2023-10-10T05:30:12Z)
• Towards Automatic Generation of Amplified Regression Test Oracles [44.45138073080198]
  We propose a test oracle derivation approach to amplify regression test oracles. The approach monitors the object state during test execution and compares it to the previous version to detect any changes in relation to the SUT's intended behaviour.
  arXiv  Detail & Related papers  (2023-07-28T12:38:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.