On Mutation-Guided Unit Test Generation

• URL: http://arxiv.org/abs/2506.02954v3
• Date: Thu, 12 Jun 2025 19:14:39 GMT
• Title: On Mutation-Guided Unit Test Generation
• Authors: Guancheng Wang, Qinghua Xu, Lionel C. Briand, Kui Liu,
• Abstract summary: MUTGEN is a mutation-guided, LLM-based test generation approach.<n>It significantly outperforms both EvoSuite and vanilla prompt-based strategies in terms of mutation score.
• Score: 9.938579776227506
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Unit tests play a vital role in uncovering potential faults in software. While tools like EvoSuite focus on maximizing code coverage, recent advances in large language models (LLMs) have shifted attention toward LLM-based test generation. However, code coverage metrics -- such as line and branch coverage -- remain overly emphasized in reported research, despite being weak indicators of a test suite's fault-detection capability. In contrast, mutation score offers a more reliable and stringent measure, as demonstrated in our findings where some test suites achieve 100% coverage but only 4% mutation score. Although a few studies consider mutation score, the effectiveness of LLMs in killing mutants remains underexplored. In this paper, we propose MUTGEN, a mutation-guided, LLM-based test generation approach that incorporates mutation feedback directly into the prompt. Evaluated on 204 subjects from two benchmarks, MUTGEN significantly outperforms both EvoSuite and vanilla prompt-based strategies in terms of mutation score. Furthermore, MUTGEN introduces an iterative generation mechanism that pushes the limits of LLMs in killing additional mutants. Our study also provide insights into the limitations of LLM-based generation, analyzing the reasons for live and uncovered mutants, and the impact of different mutation operators on generation effectiveness.

Related papers

• LLAMA: Multi-Feedback Smart Contract Fuzzing Framework with LLM-Guided Seed Generation [56.84049855266145]
  We propose a Multi-feedback Smart Contract Fuzzing framework (LLAMA) that integrates evolutionary mutation strategies, and hybrid testing techniques.<n>LLAMA achieves 91% instruction coverage and 90% branch coverage, while detecting 132 out of 148 known vulnerabilities.<n>These results highlight LLAMA's effectiveness, adaptability, and practicality in real-world smart contract security testing scenarios.
  arXiv  Detail & Related papers  (2025-07-16T09:46:58Z)
• PRIMG : Efficient LLM-driven Test Generation Using Mutant Prioritization [0.0]
  PRIMG (Prioritization and Refinement Integrated Mutation-driven Generation) is a novel framework for incremental and adaptive test case generation for Solidity smart contracts.<n> PRIMG integrates a mutation prioritization module, which employs a machine learning model trained on mutant subsumption graphs to predict the usefulness of surviving mutants.<n>The prioritization module consistently outperformed random mutant selection, enabling the generation of high-impact tests with reduced computational effort.
  arXiv  Detail & Related papers  (2025-05-08T18:30:22Z)
• Mutation Testing via Iterative Large Language Model-Driven Scientific Debugging [10.334617290353192]
  We evaluate whether Scientific computation can help Large Language Models (LLMs) to generate tests for mutants.<n>LLMs consistently outperform Pynguin in generating tests with better fault detection and coverage.<n> Importantly, we observe that the iterative refinement of test cases is important for achieving high-quality test suites.
  arXiv  Detail & Related papers  (2025-03-11T08:47:13Z)
• Beyond Binary: Towards Fine-Grained LLM-Generated Text Detection via Role Recognition and Involvement Measurement [51.601916604301685]
  Large language models (LLMs) generate content that can undermine trust in online discourse.<n>Current methods often focus on binary classification, failing to address the complexities of real-world scenarios like human-LLM collaboration.<n>To move beyond binary classification and address these challenges, we propose a new paradigm for detecting LLM-generated content.
  arXiv  Detail & Related papers  (2024-10-18T08:14:10Z)
• Large Language Models for Equivalent Mutant Detection: How Far Are We? [9.126998558502914]
  We conduct an empirical study on 3,302 method-level Java mutant pairs to investigate the effectiveness and efficiency of large language models (LLMs) for equivalent mutant detection. Our findings demonstrate that LLM-based techniques significantly outperform existing techniques, with the fine-tuned code embedding strategy being the most effective.
  arXiv  Detail & Related papers  (2024-08-03T11:58:16Z)
• Exploring Automatic Cryptographic API Misuse Detection in the Era of LLMs [60.32717556756674]
  This paper introduces a systematic evaluation framework to assess Large Language Models in detecting cryptographic misuses. Our in-depth analysis of 11,940 LLM-generated reports highlights that the inherent instabilities in LLMs can lead to over half of the reports being false positives. The optimized approach achieves a remarkable detection rate of nearly 90%, surpassing traditional methods and uncovering previously unknown misuses in established benchmarks.
  arXiv  Detail & Related papers  (2024-07-23T15:31:26Z)
• Model Surgery: Modulating LLM's Behavior Via Simple Parameter Editing [63.20133320524577]
  We show that editing a small subset of parameters can effectively modulate specific behaviors of large language models (LLMs)<n>Our approach achieves reductions of up to 90.0% in toxicity on the RealToxicityPrompts dataset and 49.2% on ToxiGen.
  arXiv  Detail & Related papers  (2024-07-11T17:52:03Z)
• An Exploratory Study on Using Large Language Models for Mutation Testing [32.91472707292504]
  Large Language Models (LLMs) have shown great potential in code-related tasks but their utility in mutation testing remains unexplored. This paper investigates the performance of LLMs in generating effective mutations to their usability, fault detection potential, and relationship with real bugs. We find that compared to existing approaches, LLMs generate more diverse mutations that are behaviorally closer to real bugs.
  arXiv  Detail & Related papers  (2024-06-14T08:49:41Z)
• An Empirical Evaluation of Manually Created Equivalent Mutants [54.02049952279685]
  Less than 10 % of manually created mutants are equivalent. Surprisingly, our findings indicate that a significant portion of developers struggle to accurately identify equivalent mutants.
  arXiv  Detail & Related papers  (2024-04-14T13:04:10Z)
• Detoxifying Large Language Models via Knowledge Editing [57.0669577257301]
  This paper investigates using knowledge editing techniques to detoxify Large Language Models (LLMs) We construct a benchmark, SafeEdit, which covers nine unsafe categories with various powerful attack prompts. We conduct experiments with several knowledge editing approaches, indicating that knowledge editing has the potential to detoxify LLMs with a limited impact on general performance efficiently.
  arXiv  Detail & Related papers  (2024-03-21T15:18:30Z)
• Self-Evaluation Improves Selective Generation in Large Language Models [54.003992911447696]
  We reformulate open-ended generation tasks into token-level prediction tasks. We instruct an LLM to self-evaluate its answers. We benchmark a range of scoring methods based on self-evaluation.
  arXiv  Detail & Related papers  (2023-12-14T19:09:22Z)
• ReEval: Automatic Hallucination Evaluation for Retrieval-Augmented Large Language Models via Transferable Adversarial Attacks [91.55895047448249]
  This paper presents ReEval, an LLM-based framework using prompt chaining to perturb the original evidence for generating new test cases. We implement ReEval using ChatGPT and evaluate the resulting variants of two popular open-domain QA datasets. Our generated data is human-readable and useful to trigger hallucination in large language models.
  arXiv  Detail & Related papers  (2023-10-19T06:37:32Z)
• Effective Test Generation Using Pre-trained Large Language Models and Mutation Testing [13.743062498008555]
  We introduce MuTAP for improving the effectiveness of test cases generated by Large Language Models (LLMs) in terms of revealing bugs. MuTAP is capable of generating effective test cases in the absence of natural language descriptions of the Program Under Test (PUTs) Our results show that our proposed method is able to detect up to 28% more faulty human-written code snippets.
  arXiv  Detail & Related papers  (2023-08-31T08:48:31Z)
• MuRS: Mutant Ranking and Suppression using Identifier Templates [4.9205581820379765]
  Google's mutation testing service integrates diff-based mutation testing into the code review process. Google's mutation testing service implements a number of suppression rules, which target not-useful mutants. This paper proposes and evaluates MuRS, an automated approach that groups mutants by patterns in the source code under test.
  arXiv  Detail & Related papers  (2023-06-15T13:43:52Z)
• Mutation Testing of Deep Reinforcement Learning Based on Real Faults [11.584571002297217]
  This paper builds on the existing approach of Mutation Testing (MT) to extend it to Reinforcement Learning (RL) systems. We show that the design choice of the mutation killing definition can affect whether or not a mutation is killed as well as the generated test cases.
  arXiv  Detail & Related papers  (2023-01-13T16:45:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.