Humanizing Automatically Generated Unit Test Suites with LLM-Based Refactoring

• URL: http://arxiv.org/abs/2606.28229v2
• Date: Wed, 01 Jul 2026 13:24:38 GMT
• Title: Humanizing Automatically Generated Unit Test Suites with LLM-Based Refactoring
• Abstract summary: Search-based test generation tools such as EvoSuite produce compilable and high-coverage unit tests at scale, but their suites are often hard to read and maintain.<n>We introduce TestHumanizer, a hybrid SBST+LLM approach that uses LLMs as controlled layers over compilable SBST suites to improve naming, structure, and developer-oriented clarity.<n>TestHumanizer reaches 88-98% compilation rates, close to EvoSuite's 100% baseline and clearly above direct generation.
• Score: 18.727291516223115
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Search-based test generation tools such as EvoSuite produce compilable and high-coverage unit tests at scale, but their suites are often hard to read and maintain. LLMs can generate more natural tests, yet direct generation remains brittle, with compilation rates of only 51-78% in our study. We introduce TestHumanizer, a hybrid SBST+LLM approach that uses LLMs as controlled refactoring layers over compilable SBST suites to improve naming, structure, and developer-oriented clarity while preserving behavior and compilation validity. We evaluate TestHumanizer on 350 classes from Defects4J and SF110. EvoSuite generates 15 suites per class, and each suite is refactored under three context configurations using gpt-4o and mistral-large-2407, yielding 31,500 refactorings. TestHumanizer reaches 88-98% compilation rates, close to EvoSuite's 100% baseline and clearly above direct LLM generation. Structural coverage is largely preserved, typically within 1-2 percentage points, and 86-95% of refactorings satisfy a composite faithful-refactoring threshold. Refactored suites also improve predicted readability, reduce control-flow and cognitive complexity, and mitigate structural smells. The summary-based setting offers the most robust trade-off, while long code-centric prompts are more prone to hallucination-induced failures. A developer study on 30 classes and 444 test methods confirms significant gains in perceived readability and willingness to adopt, with Wilcoxon p less than 0.01 and substantial inter-rater agreement. Overall, LLMs are most effective not as standalone generators but as validation-gated refinement layers over robust SBST outputs.

Related papers

• Enhancing Automated Unit Test Generation for NLP Libraries Using Large Language Models [15.717183301606061]
  Large Language Models can generate domain-relevant test code, but tests produced by LLMs alone often fail to compile or achieve sufficient coverage.<n>We propose LLMSuite, a hybrid test generation framework that integrates self-refinement prompting with class-level LLM reasoning.
  arXiv  Detail & Related papers  (2026-09-13T20:53:41Z)
• SCDBench: A Benchmark for LLM-Based Smart Contract Decompilers [55.39407031861402]
  We introduce SCDBench, a dataset and benchmark methodology for smart contract decompilation.<n>The dataset contains 600 real-world Solidity contracts with paired bytecode inputs, ground-truth source code, and replayable semantic checkpoints.<n>We evaluate Claude Opus 4.7, GPT-5.3-Codex, and GLM-5 in a zero-shot decompilation setting, including GLM-5 variants with and without extended reasoning.
  arXiv  Detail & Related papers  (2026-05-27T20:08:47Z)
• Constraint-Guided Multi-Agent Decompilation for Executable Binary Recovery [47.704311990064554]
  Decompilation is essential for security analysis, malware reverse engineering, and legacy software maintenance.<n>We present a multi-agent framework that transforms decompiled code into re-executable source through Multi-level Constraint-Guided Decompilation (MCGD)<n>Our framework achieves 84-97% re-executability, improving baseline decompiler output by 28-89 percentage points.
  arXiv  Detail & Related papers  (2026-04-27T01:28:11Z)
• HCRE: LLM-based Hierarchical Classification for Cross-Document Relation Extraction with a Prediction-then-Verification Strategy [54.91468501159335]
  Cross-document relation extraction (RE) aims to identify relations between the head tail entities located in different documents.<n>We propose an inference strategy that improves reliability through multi-view verification at each level.
  arXiv  Detail & Related papers  (2026-04-09T07:55:27Z)
• TestDecision: Sequential Test Suite Generation via Greedy Optimization and Reinforcement Learning [17.157244731153554]
  We formalize test suite generation as a MDP and demonstrate that its objective exhibits monotone submodularity.<n>We propose TestDecision, which transforms LLMs into neural greedy experts.<n>TestDecision significantly outperforms existing advanced methods.
  arXiv  Detail & Related papers  (2026-04-02T09:13:52Z)
• SWE-Refactor: A Repository-Level Benchmark for Real-World LLM-Based Code Refactoring [20.694251041823097]
  Large Language Models (LLMs) have attracted wide interest for tackling software engineering tasks.<n>Existing benchmarks commonly suffer from three shortcomings.<n>SWE-Refactor comprises 1,099 developer-written, behavior-preserving LLMs mined from 18 Java projects.
  arXiv  Detail & Related papers  (2026-02-03T16:36:29Z)
• Prism: Efficient Test-Time Scaling via Hierarchical Search and Self-Verification for Discrete Diffusion Language Models [96.0074341403456]
  Inference-time compute has re-emerged as a practical way to improve LLM reasoning.<n>Most test-time scaling (TTS) algorithms rely on autoregressive decoding.<n>We propose Prism, an efficient TTS framework for dLLMs.
  arXiv  Detail & Related papers  (2026-02-02T09:14:51Z)
• From Human to Machine Refactoring: Assessing GPT-4's Impact on Python Class Quality and Readability [46.83143241367452]
  Refactoring aims to improve code quality without altering program behavior.<n>Recent advances in Large Language Models (LLMs) have introduced new opportunities for automated code preservation.<n>We present an empirical study on LLM-driven classes using GPT-4o, applied to 100 Python classes from the ClassEval benchmark.<n>Our findings show that GPT-4o generally produces behavior-preservings that reduce code smells and improve quality metrics, albeit at the cost of decreased readability.
  arXiv  Detail & Related papers  (2026-01-19T15:22:37Z)
• Rethinking Cognitive Complexity for Unit Tests: Toward a Readability-Aware Metric Grounded in Developer Perception [15.677544288705883]
  We introduce CCTR, a Test-Aware Cognitive Complexity metric tailored for unit tests.<n>We evaluate 15,750 test suites generated by EvoSuite, GPT-4o, and Mistral Large-1024 across 350 classes from Defects4J and SF110.
  arXiv  Detail & Related papers  (2025-06-07T11:16:13Z)
• Enhancing LLM-Based Code Generation with Complexity Metrics: A Feedback-Driven Approach [6.289275189295223]
  We investigate the relationship between code complexity and the success of Large Language Models generated code.<n>We propose an iterative feedback method, where LLMs are prompted to generate correct code based on complexity metrics from previous failed outputs.<n>Experiment results show that our approach makes notable improvements, particularly with a smaller LLM.
  arXiv  Detail & Related papers  (2025-05-29T19:06:14Z)
• MANTRA: Enhancing Automated Method-Level Refactoring with Contextual RAG and Multi-Agent LLM Collaboration [44.75848695076576]
  We introduce MANTRA, a comprehensive Large Language Models agent-based framework.<n>ManTRA integrates Context-Aware Retrieval-Augmented Generation, coordinated Multi-Agent Collaboration, and Verbal Reinforcement Learning.<n> Experimental results demonstrate that MANTRA substantially surpasses a baseline LLM model.
  arXiv  Detail & Related papers  (2025-03-18T15:16:51Z)
• Test smells in LLM-Generated Unit Tests [16.061139428298986]
  This paper presents the first multi-benchmark, large-scale analysis of test smell diffusion in LLM-generated unit tests.<n>Our study draws on 20,505 class-level suites from four LLMs (GPT-3.5, GPT-4, Mistral 7B, Mixtral 8x7B), 972 method-level cases from TestBench, 14,469 EvoSuite tests, and 779,585 human-written tests from 34,635 open-source Java projects.
  arXiv  Detail & Related papers  (2024-10-14T15:35:44Z)
• LLMs as Factual Reasoners: Insights from Existing Benchmarks and Beyond [135.8013388183257]
  We propose a new protocol for inconsistency detection benchmark creation and implement it in a 10-domain benchmark called SummEdits. Most LLMs struggle on SummEdits, with performance close to random chance. The best-performing model, GPT-4, is still 8% below estimated human performance.
  arXiv  Detail & Related papers  (2023-05-23T21:50:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.