An empirical study of LoRA-based fine-tuning of large language models for automated test case generation

• URL: http://arxiv.org/abs/2604.06946v1
• Date: Wed, 08 Apr 2026 11:09:49 GMT
• Title: An empirical study of LoRA-based fine-tuning of large language models for automated test case generation
• Abstract summary: We present a comprehensive empirical study on the use of parameter-efficient fine-tuning, specifically LoRA, for requirement-based test case generation.<n>We evaluate multiple LLM families, including open-source and proprietary models, under a unified experimental pipeline.<n> Experimental results demonstrate that LoRA-based fine-tuning significantly improves the performance of all open-source models.
• Score: 5.182076496564637
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Automated test case generation from natural language requirements remains a challenging problem in software engineering due to the ambiguity of requirements and the need to produce structured, executable test artifacts. Recent advances in LLMs have shown promise in addressing this task; however, their effectiveness depends on task-specific adaptation and efficient fine-tuning strategies. In this paper, we present a comprehensive empirical study on the use of parameter-efficient fine-tuning, specifically LoRA, for requirement-based test case generation. We evaluate multiple LLM families, including open-source and proprietary models, under a unified experimental pipeline. The study systematically explores the impact of key LoRA hyperparameters, including rank, scaling factor, and dropout, on downstream performance. We propose an automated evaluation framework based on GPT-4o, which assesses generated test cases across nine quality dimensions. Experimental results demonstrate that LoRA-based fine-tuning significantly improves the performance of all open-source models, with Ministral-8B achieving the best results among them. Furthermore, we show that a fine-tuned 8B open-source model can achieve performance comparable to pre-fine-tuned GPT-4.1 models, highlighting the effectiveness of parameter-efficient adaptation. While GPT-4.1 models achieve the highest overall performance, the performance gap between proprietary and open-source models is substantially reduced after fine-tuning. These findings provide important insights into model selection, fine-tuning strategies, and evaluation methods for automated test generation. In particular, they demonstrate that cost-efficient, locally deployable open-source models can serve as viable alternatives to proprietary systems when combined with well-designed fine-tuning approaches.

Related papers

• Improving LLM-Generated Process Model Quality Through Reinforcement Learning: The Role of Reward Function Design [43.328395470827665]
  Large language models (LLMs) can generate BPMN process models from natural-language descriptions.<n>Supervised fine-tuning (SFT) limits their output quality to the patterns present in the training data.<n>We present a systematic investigation of reward function design for RL-based process model generation.
  arXiv  Detail & Related papers  (2026-07-07T11:53:02Z)
• Data-Centric Benchmarking of Exploit Generation in LLMs: Understanding the Impact of Fine-Tuning [2.491443411718293]
  We study the task of CVE-conditioned exploit generation, where a model drafts proof-of-concept exploits given software vulnerability context.<n>We adopt a data-centric approach, constructing a high-quality dataset via multi-stage preprocessing.<n>We benchmark 17 large language models across 8 evaluation criteria, providing systematic insights into their zero-shot capabilities.
  arXiv  Detail & Related papers  (2026-06-13T05:28:20Z)
• From Insight to Action: A Novel Framework for Interpretability-Guided Data Selection in Large Language Models [73.72877445629383]
  Interpretability-Guided Data Selection (IGDS) is a framework that first identifies causal task features through frequency recall and interventional filtering.<n>We validate IGDS on mathematical reasoning, summarization, and translation tasks within Gemma-2, LLaMA-3.1, and Qwen3 models.
  arXiv  Detail & Related papers  (2026-04-28T03:16:24Z)
• One Model to Critique Them All: Rewarding Agentic Tool-Use via Efficient Reasoning [54.580646706013965]
  Reward models (RMs) play a critical role in aligning large language models with human preferences.<n>We introduce ToolRM, a family of lightweight generative RMs tailored for general tool-use scenarios.<n>To build these models, we propose a novel pipeline that constructs pairwise preference data using rule-based scoring and multidimensional sampling.
  arXiv  Detail & Related papers  (2025-10-30T06:08:27Z)
• Large Language Models for Fault Localization: An Empirical Study [3.2111987440830974]
  This paper presents a systematic empirical study on large language models (LLMs) in the statement-level code fault localization task.<n>We evaluate open-source models (Qwen2.5-coder-32b-instruct, DeepSeek-V3) and closed-source models (GPT-4.1 mini, Gemini-2.5-flash) to assess their fault localization capabilities.
  arXiv  Detail & Related papers  (2025-10-23T13:04:22Z)
• Self-Improving LLM Agents at Test-Time [49.9396634315896]
  One paradigm of language model (LM) fine-tuning relies on creating large training datasets.<n>In practice, gathering large sets of data is inefficient, and training on them is prohibitively expensive.<n>We study two variants of this approach: Test-Time Self-Improvement (TT-SI) and Test-Time Distillation (TT-D)
  arXiv  Detail & Related papers  (2025-10-09T06:37:35Z)
• TCPO: Thought-Centric Preference Optimization for Effective Embodied Decision-making [75.29820290660065]
  This paper proposes Thought-Centric Preference Optimization ( TCPO) for effective embodied decision-making.<n>It emphasizes the alignment of the model's intermediate reasoning process, mitigating the problem of model degradation.<n>Experiments in the ALFWorld environment demonstrate an average success rate of 26.67%, achieving a 6% improvement over RL4VLM.
  arXiv  Detail & Related papers  (2025-09-10T11:16:21Z)
• Language Models are Few-Shot Graders [0.12289361708127876]
  We present an ASAG pipeline leveraging state-of-the-art LLMs.<n>We compare the grading performance of three OpenAI models: GPT-4, GPT-4o, and o1-preview.<n>Our findings indicate that providing graded examples enhances grading accuracy, with RAG-based selection outperforming random selection.
  arXiv  Detail & Related papers  (2025-02-18T23:38:21Z)
• SELF: Surrogate-light Feature Selection with Large Language Models in Deep Recommender Systems [51.09233156090496]
  SurrogatE-Light Feature selection method for deep recommender systems.<n> SELF integrates semantic reasoning from Large Language Models with task-specific learning from surrogate models.<n> Comprehensive experiments on three public datasets from real-world recommender platforms validate the effectiveness of SELF.
  arXiv  Detail & Related papers  (2024-12-11T16:28:18Z)
• Automatic Evaluation for Text-to-image Generation: Task-decomposed Framework, Distilled Training, and Meta-evaluation Benchmark [62.58869921806019]
  We propose a task decomposition evaluation framework based on GPT-4o to automatically construct a new training dataset. We design innovative training strategies to effectively distill GPT-4o's evaluation capabilities into a 7B open-source MLLM, MiniCPM-V-2.6. Experimental results demonstrate that our distilled open-source MLLM significantly outperforms the current state-of-the-art GPT-4o-base baseline.
  arXiv  Detail & Related papers  (2024-11-23T08:06:06Z)
• AI based Multiagent Approach for Requirements Elicitation and Analysis [3.9422957660677476]
  This study empirically investigates the effectiveness of utilizing Large Language Models (LLMs) to automate requirements analysis tasks. We deployed four models, namely GPT-3.5, GPT-4 Omni, LLaMA3-70, and Mixtral-8B, and conducted experiments to analyze requirements on four real-world projects. Preliminary results indicate notable variations in task completion among the models.
  arXiv  Detail & Related papers  (2024-08-18T07:23:12Z)
• Scaling Down to Scale Up: A Cost-Benefit Analysis of Replacing OpenAI's LLM with Open Source SLMs in Production [3.41402911469979]
  Many companies use large language models (LLMs) offered as a service, like OpenAI's GPT-4, to create AI-enabled product experiences. At the same time, a flurry of open-source small language models (SLMs) has been made available for commercial use. This paper presents a systematic evaluation methodology and a characterization of modern open-source SLMs.
  arXiv  Detail & Related papers  (2023-12-20T19:27:59Z)
• CritiqueLLM: Towards an Informative Critique Generation Model for Evaluation of Large Language Model Generation [87.44350003888646]
  Eval-Instruct can acquire pointwise grading critiques with pseudo references and revise these critiques via multi-path prompting. CritiqueLLM is empirically shown to outperform ChatGPT and all the open-source baselines.
  arXiv  Detail & Related papers  (2023-11-30T16:52:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.