Context-Enhanced LLM-Based Framework for Automatic Test Refactoring

• URL: http://arxiv.org/abs/2409.16739v1
• Date: Wed, 25 Sep 2024 08:42:29 GMT
• Title: Context-Enhanced LLM-Based Framework for Automatic Test Refactoring
• Authors: Yi Gao, Xing Hu, Xiaohu Yang, Xin Xia,
• Abstract summary: Test smells arise from poor design practices and insufficient domain knowledge. We propose UTRefactor, a context-enhanced, LLM-based framework for automatic test in Java projects. We evaluate UTRefactor on 879 tests from six open-source Java projects, reducing the number of test smells from 2,375 to 265, achieving an 89% reduction.
• Score: 10.847400457238423
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Test smells arise from poor design practices and insufficient domain knowledge, which can lower the quality of test code and make it harder to maintain and update. Manually refactoring test smells is time-consuming and error-prone, highlighting the necessity for automated approaches. Current rule-based refactoring methods often struggle in scenarios not covered by predefined rules and lack the flexibility needed to handle diverse cases effectively. In this paper, we propose a novel approach called UTRefactor, a context-enhanced, LLM-based framework for automatic test refactoring in Java projects. UTRefactor extracts relevant context from test code and leverages an external knowledge base that includes test smell definitions, descriptions, and DSL-based refactoring rules. By simulating the manual refactoring process through a chain-of-thought approach, UTRefactor guides the LLM to eliminate test smells in a step-by-step process, ensuring both accuracy and consistency throughout the refactoring. Additionally, we implement a checkpoint mechanism to facilitate comprehensive refactoring, particularly when multiple smells are present. We evaluate UTRefactor on 879 tests from six open-source Java projects, reducing the number of test smells from 2,375 to 265, achieving an 89% reduction. UTRefactor outperforms direct LLM-based refactoring methods by 61.82% in smell elimination and significantly surpasses the performance of a rule-based test smell refactoring tool. Our results demonstrate the effectiveness of UTRefactor in enhancing test code quality while minimizing manual involvement.

Related papers

• An Empirical Study on the Potential of LLMs in Automated Software Refactoring [9.157968996300417]
  We investigate the potential of large language models (LLMs) in automated software. We find that 13 out of the 176 solutions suggested by ChatGPT and 9 out of the 137 solutions suggested by Gemini were unsafe in that they either changed the functionality of the source code or introduced syntax errors.
  arXiv  Detail & Related papers  (2024-11-07T05:35:55Z)
• FactorLLM: Factorizing Knowledge via Mixture of Experts for Large Language Models [50.331708897857574]
  We introduce FactorLLM, a novel approach that decomposes well-trained dense FFNs into sparse sub-networks without requiring any further modifications. FactorLLM achieves comparable performance to the source model securing up to 85% model performance while obtaining over a 30% increase in inference speed.
  arXiv  Detail & Related papers  (2024-08-15T16:45:16Z)
• Re-ReST: Reflection-Reinforced Self-Training for Language Agents [101.22559705696885]
  Self-training in language agents can generate supervision from the agent itself. We present Reflection-Reinforced Self-Training (Re-ReST), which uses a textitreflector to refine low-quality generated samples.
  arXiv  Detail & Related papers  (2024-06-03T16:21:38Z)
• Detecting Refactoring Commits in Machine Learning Python Projects: A Machine Learning-Based Approach [3.000496428347787]
  MLRefScanner identifies commits with both ML-specific and general operations. Our study highlights the potential of ML-driven approaches in detecting programming across diverse languages and technical domains.
  arXiv  Detail & Related papers  (2024-04-09T18:46:56Z)
• Together We Go Further: LLMs and IDE Static Analysis for Extract Method Refactoring [9.882903340467815]
  Long methods that encapsulate multiple responsibilities within a single method are challenging to maintain. Large Language Models (LLMs) have been trained on large code corpora. LLMs are very effective for giving expert suggestions, yet they are unreliable: up to 76.3% of the suggestions are hallucinations.
  arXiv  Detail & Related papers  (2024-01-27T05:01:03Z)
• RefBERT: A Two-Stage Pre-trained Framework for Automatic Rename Refactoring [57.8069006460087]
  We study automatic rename on variable names, which is considered more challenging than other rename activities. We propose RefBERT, a two-stage pre-trained framework for rename on variable names. We show that the generated variable names of RefBERT are more accurate and meaningful than those produced by the existing method.
  arXiv  Detail & Related papers  (2023-05-28T12:29:39Z)
• Do code refactorings influence the merge effort? [80.1936417993664]
  Multiple contributors frequently change the source code in parallel to implement new features, fix bugs, existing code, and make other changes. These simultaneous changes need to be merged into the same version of the source code. Studies show that 10 to 20 percent of all merge attempts result in conflicts, which require the manual developer's intervention to complete the process.
  arXiv  Detail & Related papers  (2023-05-10T13:24:59Z)
• Supporting single responsibility through automated extract method refactoring [48.0838443679187]
  We propose a backward slicing algorithm to decompose a long method into slightly overlapping slices. The slices are computed for each output instruction, representing the outcome of a responsibility delegated to the method. On average, our experiments demonstrate at least a 29.6% improvement in precision and a 12.1% improvement in the recall of uncovering opportunities.
  arXiv  Detail & Related papers  (2023-05-05T11:02:02Z)
• Reinforcement Learning for Branch-and-Bound Optimisation using Retrospective Trajectories [72.15369769265398]
  Machine learning has emerged as a promising paradigm for branching. We propose retro branching; a simple yet effective approach to RL for branching. We outperform the current state-of-the-art RL branching algorithm by 3-5x and come within 20% of the best IL method's performance on MILPs with 500 constraints and 1000 variables.
  arXiv  Detail & Related papers  (2022-05-28T06:08:07Z)
• How We Refactor and How We Document it? On the Use of Supervised Machine Learning Algorithms to Classify Refactoring Documentation [25.626914797750487]
  Refactoring is the art of improving the design of a system without altering its external behavior. This study categorizes commits into 3 categories, namely, Internal QA, External QA, and Code Smell Resolution, along with the traditional BugFix and Functional categories. To better understand our classification results, we analyzed commit messages to extract patterns that developers regularly use to describe their smells.
  arXiv  Detail & Related papers  (2020-10-26T20:33:17Z)
• Toward the Automatic Classification of Self-Affirmed Refactoring [22.27416971215152]
  Self-Affirmed Refactoring (SAR) was introduced to explore how developers document their activities in commit messages. We propose a two-step approach to first identify whether a commit describes developer-related events, then to classify it according to the common quality improvement categories. Our model is able to accurately classify commits, outperforming the pattern-based random approaches, and allowing the discovery of 40 more relevant SAR patterns.
  arXiv  Detail & Related papers  (2020-09-19T18:35:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.