Hamster: A Large-Scale Study and Characterization of Developer-Written Tests

• URL: http://arxiv.org/abs/2509.26204v1
• Date: Tue, 30 Sep 2025 13:08:23 GMT
• Title: Hamster: A Large-Scale Study and Characterization of Developer-Written Tests
• Abstract summary: We study developer-written tests for Java applications, covering 1.7 million test cases from open-source repositories.<n>Our results highlight that a vast majority of developer-written tests exhibit characteristics beyond the capabilities of current ATG tools.<n>We identify promising research directions that can help bridge the gap between current tool capabilities and more effective tool support for developer testing practices.
• Score: 44.65515600399573
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Automated test generation (ATG), which aims to reduce the cost of manual test suite development, has been investigated for decades and has produced countless techniques based on a variety of approaches: symbolic analysis, search-based, random and adaptive-random, learning-based, and, most recently, large-language-model-based approaches. However, despite this large body of research, there is still a gap in our understanding of the characteristics of developer-written tests and, consequently, in our assessment of how well ATG techniques and tools can generate realistic and representative tests. To bridge this gap, we conducted an extensive empirical study of developer-written tests for Java applications, covering 1.7 million test cases from open-source repositories. Our study is the first of its kind in studying aspects of developer-written tests that are mostly neglected in the existing literature, such as test scope, test fixtures and assertions, types of inputs, and use of mocking. Based on the characterization, we then compare existing tests with those generated by two state-of-the-art ATG tools. Our results highlight that a vast majority of developer-written tests exhibit characteristics that are beyond the capabilities of current ATG tools. Finally, based on the insights gained from the study, we identify promising research directions that can help bridge the gap between current tool capabilities and more effective tool support for developer testing practices. We hope that this work can set the stage for new advances in the field and bring ATG tools closer to generating the types of tests developers write.

Related papers

• Tangent: An Empirical Study of Testing Practices for LLM-Based Agent Applications [41.648736542377016]
  Testing levels, objectives, data patterns, test complexity, and validation strategies for agent applications remain underexplored.<n>We construct a large-scale dataset of agent applications, tools, and tests, and manually label 2,572 test methods from 240 modules.<n>We derive a taxonomy of 23 testing patterns across test fixtures, data, objectives, and assertions, and characterize tests by level (unit, module, integration)
  arXiv  Detail & Related papers  (2026-08-09T02:08:34Z)
• Sakura: An Approach for Generating Complex Tests from Natural Language Test Descriptions [46.562528718959506]
  Sakura is a framework for generating tests from natural language (NL) descriptions of developer intent.<n>Sakura decomposes NL descriptions into structured blocks and processes them using a multi-agent system.<n>Sakura significantly outperforms off-the-truth agentic tools such as Gemini CLI.
  arXiv  Detail & Related papers  (2026-05-30T04:49:10Z)
• Testing with AI Agents: An Empirical Study of Test Generation Frequency, Quality, and Coverage [1.6782288954198918]
  This study presents an empirical analysis of test generation by agent-based coding tools using the AIDev dataset.<n>We extracted 2,232 commits containing test-related changes and investigated three aspects: the frequency of test additions, the structural characteristics of the generated tests, and their impact on code coverage.<n>Our findings reveal that (i) AI authored 16.4% of all commits adding tests in real-world repositories, (ii) AI-generated test methods exhibit distinct structural patterns, featuring longer code and a higher density of assertions, and (iii) AI-generated tests contribute to code coverage comparable to human-written tests
  arXiv  Detail & Related papers  (2026-03-14T03:01:44Z)
• Targeted Test Selection Approach in Continuous Integration [34.139736599165566]
  Targeted Test Selection (T-TS) is a machine learning approach for industrial test selection.<n>On live industrial data, T-TS selects only 15% of tests, reduces execution time by $5.9times$, accelerates the pipeline by $5.6times$, and detects over 95% of test failures.
  arXiv  Detail & Related papers  (2025-09-12T14:20:51Z)
• A Survey on Code Generation with LLM-based Agents [61.474191493322415]
  Code generation agents powered by large language models (LLMs) are revolutionizing the software development paradigm.<n>LLMs are characterized by three core features.<n>This paper presents a systematic survey of the field of LLM-based code generation agents.
  arXiv  Detail & Related papers  (2025-07-31T18:17:36Z)
• Intention-Driven Generation of Project-Specific Test Cases [45.2380093475221]
  We propose IntentionTest, which generates project-specific tests given the description of validation intention.<n>We extensively evaluate IntentionTest against state-of-the-art baselines (DA, ChatTester, and EvoSuite) on 4,146 test cases from 13 open-source projects.
  arXiv  Detail & Related papers  (2025-07-28T08:35:04Z)
• Use Property-Based Testing to Bridge LLM Code Generation and Validation [38.25155484701058]
  Large Language Models (LLMs) excel at code generation, but ensuring their outputs to be functionally correct is a persistent challenge.<n>This paper introduces Property-Generated Solver, a novel framework that leverages Property-Based Testing (PBT) to validate high-level program properties.<n>Property-Generated Solver employs two collaborative LLM-based agents: a Generator dedicated to code generation and iterative refinement, and a Tester that manages the PBT life-cycle.
  arXiv  Detail & Related papers  (2025-06-23T06:01:12Z)
• Evaluating Large Language Models for the Generation of Unit Tests with Equivalence Partitions and Boundary Values [42.88667535189424]
  This research evaluates the potential of Large Language Models (LLMs) in automatically generating test cases.<n>An optimized prompt was developed, that integrates code and requirements, covering critical cases such as equivalence partitions and boundary values.<n>The results show that the effectiveness of LLMs depends on well-designed prompts, robust implementation, and precise requirements.
  arXiv  Detail & Related papers  (2025-05-14T22:22:15Z)
• Disrupting Test Development with AI Assistants [1.024113475677323]
  Generative AI-assisted coding tools like GitHub Copilot, ChatGPT, and Tabnine have significantly transformed software development. This paper analyzes how these innovations impact productivity and software test development metrics.
  arXiv  Detail & Related papers  (2024-11-04T17:52:40Z)
• LLM-based Unit Test Generation via Property Retrieval [26.906316611858518]
  Property-Based Retrieval Augmentation extends LLM-based Retrieval-Augmented Generation beyond basic vector, text similarity, and graph-based methods. Our approach considers task-specific context and introduces a tailored property retrieval mechanism. We implement this approach in a tool called APT, which sequentially performs preprocessing, property retrieval, and unit test generation.
  arXiv  Detail & Related papers  (2024-10-17T13:33:12Z)
• ASTER: Natural and Multi-language Unit Test Generation with LLMs [6.259245181881262]
  We describe a generic pipeline that incorporates static analysis to guide LLMs in generating compilable and high-coverage test cases.<n>We conduct an empirical study to assess the quality of the generated tests in terms of code coverage and test naturalness.
  arXiv  Detail & Related papers  (2024-09-04T21:46:18Z)
• Leveraging Large Language Models for Enhancing the Understandability of Generated Unit Tests [4.574205608859157]
  We introduce UTGen, which combines search-based software testing and large language models to enhance the understandability of automatically generated test cases. We observe that participants working on assignments with UTGen test cases fix up to 33% more bugs and use up to 20% less time when compared to baseline test cases.
  arXiv  Detail & Related papers  (2024-08-21T15:35:34Z)
• A System for Automated Unit Test Generation Using Large Language Models and Assessment of Generated Test Suites [1.4563527353943984]
  Large Language Models (LLMs) have been applied to various aspects of software development. We present AgoneTest: an automated system for generating test suites for Java projects.
  arXiv  Detail & Related papers  (2024-08-14T23:02:16Z)
• Leveraging Large Language Models for Efficient Failure Analysis in Game Development [47.618236610219554]
  This paper proposes a new approach to automatically identify which change in the code caused a test to fail. The method leverages Large Language Models (LLMs) to associate error messages with the corresponding code changes causing the failure. Our approach reaches an accuracy of 71% in our newly created dataset, which comprises issues reported by developers at EA over a period of one year.
  arXiv  Detail & Related papers  (2024-06-11T09:21:50Z)
• Towards Automatic Generation of Amplified Regression Test Oracles [44.45138073080198]
  We propose a test oracle derivation approach to amplify regression test oracles. The approach monitors the object state during test execution and compares it to the previous version to detect any changes in relation to the SUT's intended behaviour.
  arXiv  Detail & Related papers  (2023-07-28T12:38:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.