Leveraging Large Language Models for Enhancing the Understandability of Generated Unit Tests

• URL: http://arxiv.org/abs/2408.11710v1
• Date: Wed, 21 Aug 2024 15:35:34 GMT
• Title: Leveraging Large Language Models for Enhancing the Understandability of Generated Unit Tests
• Authors: Amirhossein Deljouyi, Roham Koohestani, Maliheh Izadi, Andy Zaidman,
• Abstract summary: We introduce UTGen, which combines search-based software testing and large language models to enhance the understandability of automatically generated test cases. We observe that participants working on assignments with UTGen test cases fix up to 33% more bugs and use up to 20% less time when compared to baseline test cases.
• Score: 4.574205608859157
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Automated unit test generators, particularly search-based software testing tools like EvoSuite, are capable of generating tests with high coverage. Although these generators alleviate the burden of writing unit tests, they often pose challenges for software engineers in terms of understanding the generated tests. To address this, we introduce UTGen, which combines search-based software testing and large language models to enhance the understandability of automatically generated test cases. We achieve this enhancement through contextualizing test data, improving identifier naming, and adding descriptive comments. Through a controlled experiment with 32 participants from both academia and industry, we investigate how the understandability of unit tests affects a software engineer's ability to perform bug-fixing tasks. We selected bug-fixing to simulate a real-world scenario that emphasizes the importance of understandable test cases. We observe that participants working on assignments with UTGen test cases fix up to 33% more bugs and use up to 20% less time when compared to baseline test cases. From the post-test questionnaire, we gathered that participants found that enhanced test names, test data, and variable names improved their bug-fixing process.

Related papers

• CLOVER: A Test Case Generation Benchmark with Coverage, Long-Context, and Verification [71.34070740261072]
  This paper presents a benchmark, CLOVER, to evaluate models' capabilities in generating and completing test cases. The benchmark is containerized for code execution across tasks, and we will release the code, data, and construction methodologies.
  arXiv  Detail & Related papers  (2025-02-12T21:42:56Z)
• Learning to Generate Unit Tests for Automated Debugging [52.63217175637201]
  Unit tests (UTs) play an instrumental role in assessing code correctness as well as providing feedback to a large language model (LLM) We propose UTGen, which teaches LLMs to generate unit test inputs that reveal errors along with their correct expected outputs. We show that UTGen outperforms UT generation baselines by 7.59% based on a metric measuring the presence of both error-revealing UT inputs and correct UT outputs.
  arXiv  Detail & Related papers  (2025-02-03T18:51:43Z)
• ViUniT: Visual Unit Tests for More Robust Visual Programming [104.55763189099125]
  When models answer correctly, they produce incorrect programs 33% of the time. We propose Visual Unit Testing (ViUniT), a framework to improve the reliability of visual programs by automatically generating unit tests.
  arXiv  Detail & Related papers  (2024-12-12T01:36:18Z)
• ASTER: Natural and Multi-language Unit Test Generation with LLMs [6.259245181881262]
  We describe a generic pipeline that incorporates static analysis to guide LLMs in generating compilable and high-coverage test cases. We conduct an empirical study to assess the quality of the generated tests in terms of code coverage and test naturalness.
  arXiv  Detail & Related papers  (2024-09-04T21:46:18Z)
• Observation-based unit test generation at Meta [52.4716552057909]
  TestGen automatically generates unit tests, carved from serialized observations of complex objects, observed during app execution. TestGen has landed 518 tests into production, which have been executed 9,617,349 times in continuous integration, finding 5,702 faults. Our evaluation reveals that, when carving its observations from 4,361 reliable end-to-end tests, TestGen was able to generate tests for at least 86% of the classes covered by end-to-end tests.
  arXiv  Detail & Related papers  (2024-02-09T00:34:39Z)
• Automatic Generation of Test Cases based on Bug Reports: a Feasibility Study with Large Language Models [4.318319522015101]
  Existing approaches produce test cases that either can be qualified as simple (e.g. unit tests) or that require precise specifications. Most testing procedures still rely on test cases written by humans to form test suites. We investigate the feasibility of performing this generation by leveraging large language models (LLMs) and using bug reports as inputs.
  arXiv  Detail & Related papers  (2023-10-10T05:30:12Z)
• Prompting Code Interpreter to Write Better Unit Tests on Quixbugs Functions [0.05657375260432172]
  Unit testing is a commonly-used approach in software engineering to test the correctness and robustness of written code. In this study, we explore the effect of different prompts on the quality of unit tests generated by Code Interpreter. We find that the quality of the generated unit tests is not sensitive to changes in minor details in the prompts provided.
  arXiv  Detail & Related papers  (2023-09-30T20:36:23Z)
• Effective Test Generation Using Pre-trained Large Language Models and Mutation Testing [13.743062498008555]
  We introduce MuTAP for improving the effectiveness of test cases generated by Large Language Models (LLMs) in terms of revealing bugs. MuTAP is capable of generating effective test cases in the absence of natural language descriptions of the Program Under Test (PUTs) Our results show that our proposed method is able to detect up to 28% more faulty human-written code snippets.
  arXiv  Detail & Related papers  (2023-08-31T08:48:31Z)
• Towards Automatic Generation of Amplified Regression Test Oracles [44.45138073080198]
  We propose a test oracle derivation approach to amplify regression test oracles. The approach monitors the object state during test execution and compares it to the previous version to detect any changes in relation to the SUT's intended behaviour.
  arXiv  Detail & Related papers  (2023-07-28T12:38:44Z)
• Unit Test Case Generation with Transformers and Focal Context [10.220204860586582]
  AthenaTest aims to generate unit test cases by learning from real-world focal methods and developer-written test cases. We introduce Methods2Test, the largest publicly available supervised parallel corpus of unit test case methods and corresponding focal methods in Java. We evaluate AthenaTest on five defects4j projects, generating 25K passing test cases covering 43.7% of the focal methods with only 30 attempts.
  arXiv  Detail & Related papers  (2020-09-11T18:57:36Z)
• Beyond Accuracy: Behavioral Testing of NLP models with CheckList [66.42971817954806]
  CheckList is a task-agnostic methodology for testing NLP models. CheckList includes a matrix of general linguistic capabilities and test types that facilitate comprehensive test ideation. In a user study, NLP practitioners with CheckList created twice as many tests, and found almost three times as many bugs as users without it.
  arXiv  Detail & Related papers  (2020-05-08T15:48:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.