LLM-based Unit Test Generation for Dynamically-Typed Programs

• URL: http://arxiv.org/abs/2503.14000v1
• Date: Tue, 18 Mar 2025 08:07:17 GMT
• Title: LLM-based Unit Test Generation for Dynamically-Typed Programs
• Authors: Runlin Liu, Zhe Zhang, Yunge Hu, Yuhang Lin, Xiang Gao, Hailong Sun,
• Abstract summary: TypeTest is a novel framework that enhances type correctness in test generation through a vector-based Retrieval-Augmented Generation system.<n>In an evaluation on 125 real-world Python modules, TypeTest achieved an average statement coverage of 86.6% and branch coverage of 76.8%, outperforming state-of-theart tools by 5.4% and 9.3%, respectively.
• Score: 16.38145000434927
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Automated unit test generation has been widely studied, but generating effective tests for dynamically typed programs remains a significant challenge. Existing approaches, including search-based software testing (SBST) and recent LLM-based methods, often suffer from type errors, leading to invalid inputs and assertion failures, ultimately reducing testing effectiveness. To address this, we propose TypeTest, a novel framework that enhances type correctness in test generation through a vector-based Retrieval-Augmented Generation (RAG) system. TypeTest employs call instance retrieval and feature-based retrieval to infer parameter types accurately and construct valid test inputs. Furthermore, it utilizes the call graph to extract richer contextual information, enabling more accurate assertion generation. In addition, TypeTest incorporates a repair mechanism and iterative test generation, progressively refining test cases to improve coverage. In an evaluation on 125 real-world Python modules, TypeTest achieved an average statement coverage of 86.6% and branch coverage of 76.8%, outperforming state-of-theart tools by 5.4% and 9.3%, respectively.

Related papers

• Learning to Generate Unit Tests for Automated Debugging [52.63217175637201]
  Unit tests (UTs) play an instrumental role in assessing code correctness as well as providing feedback to large language models (LLMs)<n>We propose UTGen, which teaches LLMs to generate unit test inputs that reveal errors along with their correct expected outputs.<n>We show that UTGen outperforms other LLM-based baselines by 7.59% based on a metric measuring the presence of both error-revealing UT inputs and correct UT outputs.
  arXiv  Detail & Related papers  (2025-02-03T18:51:43Z)
• Context-Aware Testing: A New Paradigm for Model Testing with Large Language Models [49.06068319380296]
  We introduce context-aware testing (CAT) which uses context as an inductive bias to guide the search for meaningful model failures. We instantiate the first CAT system, SMART Testing, which employs large language models to hypothesize relevant and likely failures.
  arXiv  Detail & Related papers  (2024-10-31T15:06:16Z)
• TestGenEval: A Real World Unit Test Generation and Test Completion Benchmark [24.14654309612826]
  TestGenEval comprises 68,647 tests from 1,210 code and test file pairs across 11 well-maintained Python repositories. It covers initial tests authoring, test suite completion, and code coverage improvements. We evaluate several popular models, with sizes ranging from 7B to 405B parameters.
  arXiv  Detail & Related papers  (2024-10-01T14:47:05Z)
• TestART: Improving LLM-based Unit Testing via Co-evolution of Automated Generation and Repair Iteration [7.833381226332574]
  Large language models (LLMs) have demonstrated remarkable capabilities in generating unit test cases.<n>We propose TestART, a novel unit test generation method.<n>TestART improves LLM-based unit testing via co-evolution of automated generation and repair iteration.
  arXiv  Detail & Related papers  (2024-08-06T10:52:41Z)
• GPT-HateCheck: Can LLMs Write Better Functional Tests for Hate Speech Detection? [50.53312866647302]
  HateCheck is a suite for testing fine-grained model functionalities on synthesized data. We propose GPT-HateCheck, a framework to generate more diverse and realistic functional tests from scratch. Crowd-sourced annotation demonstrates that the generated test cases are of high quality.
  arXiv  Detail & Related papers  (2024-02-23T10:02:01Z)
• Observation-based unit test generation at Meta [52.4716552057909]
  TestGen automatically generates unit tests, carved from serialized observations of complex objects, observed during app execution. TestGen has landed 518 tests into production, which have been executed 9,617,349 times in continuous integration, finding 5,702 faults. Our evaluation reveals that, when carving its observations from 4,361 reliable end-to-end tests, TestGen was able to generate tests for at least 86% of the classes covered by end-to-end tests.
  arXiv  Detail & Related papers  (2024-02-09T00:34:39Z)
• Code-Aware Prompting: A study of Coverage Guided Test Generation in Regression Setting using LLM [32.44432906540792]
  We present SymPrompt, a code-aware prompting strategy for large language models in test generation. SymPrompt enhances correct test generations by a factor of 5 and bolsters relative coverage by 26% for CodeGen2. Notably, when applied to GPT-4, SymPrompt improves coverage by over 2x compared to baseline prompting strategies.
  arXiv  Detail & Related papers  (2024-01-31T18:21:49Z)
• Effective Test Generation Using Pre-trained Large Language Models and Mutation Testing [13.743062498008555]
  We introduce MuTAP for improving the effectiveness of test cases generated by Large Language Models (LLMs) in terms of revealing bugs. MuTAP is capable of generating effective test cases in the absence of natural language descriptions of the Program Under Test (PUTs) Our results show that our proposed method is able to detect up to 28% more faulty human-written code snippets.
  arXiv  Detail & Related papers  (2023-08-31T08:48:31Z)
• Towards Automatic Generation of Amplified Regression Test Oracles [44.45138073080198]
  We propose a test oracle derivation approach to amplify regression test oracles. The approach monitors the object state during test execution and compares it to the previous version to detect any changes in relation to the SUT's intended behaviour.
  arXiv  Detail & Related papers  (2023-07-28T12:38:44Z)
• A Comprehensive Survey on Test-Time Adaptation under Distribution Shifts [117.72709110877939]
  Test-time adaptation (TTA) has the potential to adapt a pre-trained model to unlabeled data during testing, before making predictions. We categorize TTA into several distinct groups based on the form of test data, namely, test-time domain adaptation, test-time batch adaptation, and online test-time adaptation.
  arXiv  Detail & Related papers  (2023-03-27T16:32:21Z)
• An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generation [3.9762912548964864]
  This paper presents a large-scale empirical evaluation on the effectiveness of Large Language Models for automated unit test generation. We implement our approach in TestPilot, a test generation tool for JavaScript that automatically generates unit tests for all API functions in an npm package. We find that 92.8% of TestPilot's generated tests have no more than 50% similarity with existing tests.
  arXiv  Detail & Related papers  (2023-02-13T17:13:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.