An Empirical Evaluation of Using Large Language Models for Automated Model-Based Test Generation

• URL: http://arxiv.org/abs/2608.27094v1
• Date: Thu, 27 Aug 2026 13:13:12 GMT
• Title: An Empirical Evaluation of Using Large Language Models for Automated Model-Based Test Generation
• Abstract summary: This paper presents an empirical evaluation of Large Language Models (LLMs) for automated model-based test generation.<n>Our evaluation indicates strong potential to optimize and shorten test paths and step sizes using the recent five state-of-the-art LLMs.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models have shown strong potential for software engineering tasks, particularly software testing. Model-based testing (MBT) is a software testing technique. To address the broad scalability challenge for industrial adoption of MBTs, our paper presents an empirical evaluation of Large Language Models (LLMs) for automated model-based test generation, compared with a state-of-the-art model-based testing tool (GraphWalker) and its built-in algorithms (random and quick random for edge and vertex coverage settings). Our evaluation indicates strong potential to optimize and shorten test paths and step sizes using the recent five state-of-the-art LLMs (GPT-5.1, GPT-5.2, Claude Opus 4.5, Claude Sonnet 4.5, and Gemini 2.5 Pro) against four GraphWalker models (two web applications (Parabank and Testinium) and two hardware applications (TLC and RISC-V) ) of escalating complexity.

Related papers

• Exploring LLM-based Verilog Code Generation with Data-Efficient Fine-Tuning and Testbench Automation [1.2868785387618027]
  This paper presents a workflow that uses multi-agent models to generate testbenches for high-quality fine-tuning data.<n>By automating testbench creation, the fine-tuned model for the specification-to-Verilog task achieves performance comparable to state-of-the-art methods.
  arXiv  Detail & Related papers  (2026-04-16T06:02:39Z)
• Prompting Strategies for Language Model-Based Item Generation in K-12 Education: Bridging the Gap Between Small and Large Language Models [5.584522240405349]
  This study explores automatic generation (AIG) using language models to create multiple choice questions (MCQs) for morphological assessment.<n>We evaluated seven structured prompting strategies, including zero-shot, few-shot, chain-of-thought, role-based, sequential, and combinations.<n>Results show that structured prompting, especially strategies combining chain-of-thought and sequential design, significantly improved Gemma's outputs.
  arXiv  Detail & Related papers  (2025-08-27T18:54:32Z)
• Technical Report: Full-Stack Fine-Tuning for the Q Programming Language [1.2316583133621197]
  We release an evaluation dataset for the Q programming language.<n>We benchmark major frontier models on the dataset, then do pretraining, supervised fine tuning, and reinforcement learning.<n>Our best model achieves a pass@1 accuracy of 59 percent on our Q benchmark, surpassing the best-performing frontier model.
  arXiv  Detail & Related papers  (2025-08-09T04:22:07Z)
• MBTModelGenerator: A software tool for reverse engineering of Model-based Testing (MBT) models from clickstream data of web applications [1.516251872371896]
  The tool captures UI events, transforms them into state-transition models, and exports the result in a format compatible with the GraphWalker MBT tool.<n>This report documents the system requirements, design decisions, implementation details, testing process, and empirical evaluation of the tool, which is publicly available as open-source.
  arXiv  Detail & Related papers  (2025-06-09T19:44:10Z)
• Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling [191.7830199016589]
  We introduce InternVL 2.5, an advanced multimodal large language model (MLLM) series that builds upon InternVL 2.0.<n>InternVL 2.5 exhibits competitive performance, rivaling leading commercial models such as GPT-4o and Claude-3.5-Sonnet.<n>We hope this model contributes to the open-source community by setting new standards for developing and applying multimodal AI systems.
  arXiv  Detail & Related papers  (2024-12-06T18:57:08Z)
• Promises and Pitfalls of Generative Masked Language Modeling: Theoretical Framework and Practical Guidelines [74.42485647685272]
  We focus on Generative Masked Language Models (GMLMs) We train a model to fit conditional probabilities of the data distribution via masking, which are subsequently used as inputs to a Markov Chain to draw samples from the model. We adapt the T5 model for iteratively-refined parallel decoding, achieving 2-3x speedup in machine translation with minimal sacrifice in quality.
  arXiv  Detail & Related papers  (2024-07-22T18:00:00Z)
• Design2Code: Benchmarking Multimodal Code Generation for Automated Front-End Engineering [74.99736967448423]
  We construct Design2Code - the first real-world benchmark for this task.<n>We manually curate 484 diverse real-world webpages as test cases and develop a set of automatic evaluation metrics.<n>Our fine-grained break-down metrics indicate that models mostly lag in recalling visual elements from the input webpages and generating correct layout designs.
  arXiv  Detail & Related papers  (2024-03-05T17:56:27Z)
• TAT-LLM: A Specialized Language Model for Discrete Reasoning over Tabular and Textual Data [73.29220562541204]
  We consider harnessing the amazing power of language models (LLMs) to solve our task. We develop a TAT-LLM language model by fine-tuning LLaMA 2 with the training data generated automatically from existing expert-annotated datasets.
  arXiv  Detail & Related papers  (2024-01-24T04:28:50Z)
• VeriGen: A Large Language Model for Verilog Code Generation [22.837558083876743]
  We fine-tune pre-existing Large Language Models (LLMs) on Verilog datasets compiled from GitHub and Verilog textbooks. Here, our fine-tuned open-source CodeGen-16B model outperforms the commercial state-of-the-art GPT-3.5-turbo model with a 1.1% overall increase. Notably, it demonstrates a 41% improvement in generating syntactically correct Verilog code across various problem categories compared to its pre-trained counterpart.
  arXiv  Detail & Related papers  (2023-07-28T02:57:14Z)
• Large Language Models in the Workplace: A Case Study on Prompt Engineering for Job Type Classification [58.720142291102135]
  This case study investigates the task of job classification in a real-world setting. The goal is to determine whether an English-language job posting is appropriate for a graduate or entry-level position.
  arXiv  Detail & Related papers  (2023-03-13T14:09:53Z)
• Text Embeddings by Weakly-Supervised Contrastive Pre-training [98.31785569325402]
  E5 is a family of state-of-the-art text embeddings that transfer well to a wide range of tasks. E5 can be readily used as a general-purpose embedding model for any tasks requiring a single-vector representation of texts.
  arXiv  Detail & Related papers  (2022-12-07T09:25:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.