Testing Framework Migration with Large Language Models

• URL: http://arxiv.org/abs/2602.02964v1
• Date: Tue, 03 Feb 2026 01:04:09 GMT
• Title: Testing Framework Migration with Large Language Models
• Abstract summary: Python developers rely on two major testing frameworks: textttunittest and textttPytest.<n>We investigate the capability of Large Language Models (LLMs) to automate test framework migrations from textttunittest to textttPytest.
• Score: 0.29332247106953097
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Python developers rely on two major testing frameworks: \texttt{unittest} and \texttt{Pytest}. While \texttt{Pytest} offers simpler assertions, reusable fixtures, and better interoperability, migrating existing suites from \texttt{unittest} remains a manual and time-consuming process. Automating this migration could substantially reduce effort and accelerate test modernization. In this paper, we investigate the capability of Large Language Models (LLMs) to automate test framework migrations from \texttt{unittest} to \texttt{Pytest}. We evaluate GPT 4o and Claude Sonnet 4 under three prompting strategies (Zero-shot, One-shot, and Chain-of-Thought) and two temperature settings (0.0 and 1.0). To support this analysis, we first introduce a curated dataset of real-world migrations extracted from the top 100 Python open-source projects. Next, we actually execute the LLM-generated test migrations in their respective test suites. Overall, we find that 51.5% of the LLM-generated test migrations failed, while 48.5% passed. The results suggest that LLMs can accelerate test migration, but there are often caveats. For example, Claude Sonnet 4 exhibited more conservative migrations (e.g., preserving class-based tests and legacy \texttt{unittest} references), while GPT-4o favored more transformations (e.g., to function-based tests). We conclude by discussing multiple implications for practitioners and researchers.

Related papers

• XREPOTEST: Benchmarking Multilingual Repository-Level Unit Test Generation for Large Language Models [8.178653684643876]
  XREPOTEST is a repository-level benchmark for unit test generation spanning five programming languages.<n>XREPOTEST evaluates tests under realistic repository constraints using a containerized execution framework.<n>Experiments with 14 state-of-the-art LLMs, including Claude 4.5, GPT-5.2, DeepSeek V4-Pro, and Qwen families, reveal a substantial gap between standalone and repository-level performance.
  arXiv  Detail & Related papers  (2026-08-26T15:53:43Z)
• IntentTester: Intent-Driven Multi-agent Framework for Cross-Library Test Migration [14.0813614333238]
  We present IntentTester, a framework for intent-driven test reuse.<n>We evaluate IntentTester on nine open-source projects across three domains (JSON, HTML, and Time) and two languages (Java and Python)
  arXiv  Detail & Related papers  (2026-06-24T08:59:01Z)
• Improving LLM-Driven Test Generation by Learning from Mocking Information [3.7334264501860375]
  MOCKMILL generates test cases by exploiting mocking information automatically extracted from developer-written tests.<n>We evaluated MOCKMILL on 10 open-source classes from six Java projects using four Large Language Models.
  arXiv  Detail & Related papers  (2026-04-21T10:24:26Z)
• TestMigrationsInPy: A Dataset of Test Migrations from Unittest to Pytest [0.0]
  Unittest and pytest are the most popular testing frameworks in Python.<n> pytest provides some advantages, including simpler assertion, reuse of fixtures, and interoperability.<n>In this paper, we propose TestMigrationsInPy, a dataset of test migrations from unittest to pytest.
  arXiv  Detail & Related papers  (2026-02-04T23:14:17Z)
• LLMs for Automated Unit Test Generation and Assessment in Java: The AgoneTest Framework [2.501198441875755]
  AgoneTest is an evaluation framework for Large Language Model-generated unit tests in Java.<n>For the subset of tests that compile, LLM-generated tests can match or exceed human-written tests in terms of coverage and defect detection.
  arXiv  Detail & Related papers  (2025-11-25T15:33:00Z)
• GPT-4.1 Sets the Standard in Automated Experiment Design Using Novel Python Libraries [0.649540541957527]
  Large Language Models (LLMs) have advanced rapidly as tools for automating code generation in scientific research.<n>This study systematically benchmarks a selection of state-of-the-art LLMs in generating functional Python code.<n>Results show that only a small subset of models consistently generate correct, executable code.
  arXiv  Detail & Related papers  (2025-07-30T13:11:29Z)
• YATE: The Role of Test Repair in LLM-Based Unit Test Generation [22.67442101368384]
  We propose a technique for repairing some of these incorrect tests through a combination of rule-based static analysis and re-prompting.<n>We evaluate this simple approach, named YATE, on a set of 6 open-source projects.<n>YATE achieves 22% higher line coverage, 20% higher branch coverage and kill 20% more mutants at a comparable cost.
  arXiv  Detail & Related papers  (2025-07-24T11:32:31Z)
• Runaway is Ashamed, But Helpful: On the Early-Exit Behavior of Large Language Model-based Agents in Embodied Environments [54.67512489842682]
  Large language models (LLMs) have demonstrated strong planning and decision-making capabilities in complex embodied environments.<n>We take a first step toward exploring the early-exit behavior for LLM-based agents.
  arXiv  Detail & Related papers  (2025-05-23T08:23:36Z)
• EnvBench: A Benchmark for Automated Environment Setup [76.02998475135824]
  Large Language Models have enabled researchers to focus on practical repository-level tasks in software engineering domain.<n>Existing studies on environment setup introduce innovative agentic strategies, but their evaluation is often based on small datasets.<n>To address this gap, we introduce a comprehensive environment setup benchmark EnvBench.
  arXiv  Detail & Related papers  (2025-03-18T17:19:12Z)
• ProjectTest: A Project-level LLM Unit Test Generation Benchmark and Impact of Error Fixing Mechanisms [48.43237545197775]
  Unit test generation has become a promising and important use case of LLMs.<n>ProjectTest is a project-level benchmark for unit test generation covering Python, Java, and JavaScript.
  arXiv  Detail & Related papers  (2025-02-10T15:24:30Z)
• LlamaRestTest: Effective REST API Testing with Small Language Models [50.058600784556816]
  We present LlamaRestTest, a novel approach that employs two custom Large Language Models (LLMs) to generate realistic test inputs.<n>We evaluate it against several state-of-the-art REST API testing tools, including RESTGPT, a GPT-powered specification-enhancement tool.<n>Our study shows that small language models can perform as well as, or better than, large language models in REST API testing.
  arXiv  Detail & Related papers  (2025-01-15T05:51:20Z)
• Observation-based unit test generation at Meta [52.4716552057909]
  TestGen automatically generates unit tests, carved from serialized observations of complex objects, observed during app execution. TestGen has landed 518 tests into production, which have been executed 9,617,349 times in continuous integration, finding 5,702 faults. Our evaluation reveals that, when carving its observations from 4,361 reliable end-to-end tests, TestGen was able to generate tests for at least 86% of the classes covered by end-to-end tests.
  arXiv  Detail & Related papers  (2024-02-09T00:34:39Z)
• MoT: Memory-of-Thought Enables ChatGPT to Self-Improve [73.90376920653507]
  We propose a framework, Memory-of-Thought, to let the Large Language Models self-improve without annotated datasets and parameter updates. Experimental results show that MoT can help ChatGPT significantly improve its abilities in arithmetic reasoning, commonsense reasoning, factual reasoning, and natural language inference.
  arXiv  Detail & Related papers  (2023-05-09T05:25:05Z)
• An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generation [3.9762912548964864]
  This paper presents a large-scale empirical evaluation on the effectiveness of Large Language Models for automated unit test generation. We implement our approach in TestPilot, a test generation tool for JavaScript that automatically generates unit tests for all API functions in an npm package. We find that 92.8% of TestPilot's generated tests have no more than 50% similarity with existing tests.
  arXiv  Detail & Related papers  (2023-02-13T17:13:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.