Harnessing the Power of Large Language Models for Software Testing Education: A Focus on ISTQB Syllabus

• URL: http://arxiv.org/abs/2510.22318v1
• Date: Sat, 25 Oct 2025 14:45:58 GMT
• Title: Harnessing the Power of Large Language Models for Software Testing Education: A Focus on ISTQB Syllabus
• Abstract summary: The International Software Testing Qualifications Board (ISTQB) certification framework is globally recognized and widely adopted in industry and academia.<n>This paper explores and evaluates how large language models (LLMs) can complement the ISTQB framework for higher education.
• Score: 0.6524460254566903
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Software testing is a critical component in the software engineering field and is important for software engineering education. Thus, it is vital for academia to continuously improve and update educational methods to reflect the current state of the field. The International Software Testing Qualifications Board (ISTQB) certification framework is globally recognized and widely adopted in industry and academia. However, ISTQB-based learning has been rarely applied with recent generative artificial intelligence advances. Despite the growing capabilities of large language models (LLMs), ISTQB-based learning and instruction with LLMs have not been thoroughly explored. This paper explores and evaluates how LLMs can complement the ISTQB framework for higher education. The findings present four key contributions: (i) the creation of a comprehensive ISTQB-aligned dataset spanning over a decade, consisting of 28 sample exams and 1,145 questions; (ii) the development of a domain-optimized prompt that enhances LLM precision and explanation quality on ISTQB tasks; (iii) a systematic evaluation of state-of-the-art LLMs on this dataset; and (iv) actionable insights and recommendations for integrating LLMs into software testing education. These findings highlight the promise of LLMs in supporting ISTQB certification preparation and offer a foundation for their broader use in software engineering at higher education.

Related papers

• LLMs taking shortcuts in test generation: A study with SAP HANA and LevelDB [0.22940141855172033]
  Large Language Models (LLMs) have achieved impressive results on public benchmarks, often leading to claims of advanced reasoning.<n>This paper investigates behavior in automated test generation for software memorization, contrasting performance on an open-source system.<n>Results show that LLMs excel on familiar, open-source benchmarks but struggle with unseen, complex, often prioritizing compilability over effectiveness.
  arXiv  Detail & Related papers  (2026-04-15T21:30:02Z)
• Are LLMs Ready for Computer Science Education? A Cross-Domain, Cross-Lingual and Cognitive-Level Evaluation Using Professional Certification Exams [34.15435577373619]
  GPT-5 performed best on English-language certifications, while Qwen-Plus performed better in Chinese contexts.<n>DeepSeek-R1 achieved the most balanced cross-lingual performance, whereas Llama-3.3 showed clear limitations in higher-order reasoning and robustness.
  arXiv  Detail & Related papers  (2026-04-08T09:56:31Z)
• From Code Foundation Models to Agents and Applications: A Practical Guide to Code Intelligence [150.3696990310269]
  Large language models (LLMs) have transformed automated software development by enabling direct translation of natural language descriptions into functional code.<n>We provide a comprehensive synthesis and practical guide (a series of analytic and probing experiments) about code LLMs.<n>We analyze the code capability of the general LLMs (GPT-4, Claude, LLaMA) and code-specialized LLMs (StarCoder, Code LLaMA, DeepSeek-Coder, and QwenCoder)
  arXiv  Detail & Related papers  (2025-11-23T17:09:34Z)
• Benchmarking Chinese Commonsense Reasoning with a Multi-hop Reasoning Perspective [53.594353527056775]
  We propose Chinese Commonsense Multi-hop Reasoning ( CCMOR) to evaluate Large Language Models (LLMs)<n> CCMOR is designed to evaluate LLMs' ability to integrate Chinese-specific factual knowledge with multi-step logical reasoning.<n>We implement a human-in-the-loop verification system, where domain experts systematically validate and refine the generated questions.
  arXiv  Detail & Related papers  (2025-10-09T20:29:00Z)
• Using LLMs and Essence to Support Software Practice Adoption [0.3609538870261841]
  This study explores the integration of Essence, a standard and thinking framework for managing software engineering practices, with large language models (LLMs)<n>The proposed system consistently outperforms its baseline counterpart in domain-specific tasks.
  arXiv  Detail & Related papers  (2025-08-22T14:59:35Z)
• SPARQL Query Generation with LLMs: Measuring the Impact of Training Data Memorization and Knowledge Injection [81.78173888579941]
  Large Language Models (LLMs) are considered a well-suited method to increase the quality of the question-answering functionality.<n>LLMs are trained on web data, where researchers have no control over whether the benchmark or the knowledge graph was already included in the training data.<n>This paper introduces a novel method that evaluates the quality of LLMs by generating a SPARQL query from a natural-language question.
  arXiv  Detail & Related papers  (2025-07-18T12:28:08Z)
• MERA Code: A Unified Framework for Evaluating Code Generation Across Tasks [56.34018316319873]
  We propose MERA Code, a benchmark for evaluating code for the latest code generation LLMs in Russian.<n>This benchmark includes 11 evaluation tasks that span 8 programming languages.<n>We evaluate open LLMs and frontier API models, analyzing their limitations in terms of practical coding tasks in non-English languages.
  arXiv  Detail & Related papers  (2025-07-16T14:31:33Z)
• Are LLMs Ready for English Standardized Tests? A Benchmarking and Elicitation Perspective [13.167177024716338]
  Large language models (LLMs) hold promise for revolutionizing how learners interact with educational content.<n>We assess their ability to generate accurate and contextually appropriate solutions across a diverse set of English Standardized Tests (ESTs)
  arXiv  Detail & Related papers  (2025-05-17T05:10:44Z)
• Challenges in Testing Large Language Model Based Software: A Faceted Taxonomy [8.927002750209295]
  Large Language Models (LLMs) and Multi-Agent LLMs (MALLMs) introduce non-determinism unlike traditional or machine learning software.<n>This paper presents a taxonomy for LLM test case design, informed by research literature and our experience.
  arXiv  Detail & Related papers  (2025-03-01T13:15:56Z)
• Revolutionizing Database Q&A with Large Language Models: Comprehensive Benchmark and Evaluation [43.217701432032484]
  We introduce DQABench, the first comprehensive database QA benchmark for Large Language Models (LLMs)<n> DQABench features an innovative LLM-based method to automate the generation, cleaning, and rewriting of evaluation dataset, resulting in over 200,000 QA pairs in English and Chinese, separately.<n> Furthermore, we propose a comprehensive database QA testbed DQATestbed. This testbed is highly modular and scalable, with basic and advanced components such as Question Classification Routing (QCR), RAG, TIG, and Prompt template Engineering (PTE)
  arXiv  Detail & Related papers  (2024-09-05T13:45:42Z)
• Benchmarking Uncertainty Quantification Methods for Large Language Models with LM-Polygraph [83.90988015005934]
  Uncertainty quantification is a key element of machine learning applications.<n>We introduce a novel benchmark that implements a collection of state-of-the-art UQ baselines.<n>We conduct a large-scale empirical investigation of UQ and normalization techniques across eleven tasks, identifying the most effective approaches.
  arXiv  Detail & Related papers  (2024-06-21T20:06:31Z)
• A Software Engineering Perspective on Testing Large Language Models: Research, Practice, Tools and Benchmarks [2.8061460833143346]
  Large Language Models (LLMs) are rapidly becoming ubiquitous both as stand-alone tools and as components of current and future software systems. To enable usage of LLMs in the high-stake or safety-critical systems of 2030, they need to undergo rigorous testing.
  arXiv  Detail & Related papers  (2024-06-12T13:45:45Z)
• Prompting Large Language Models to Tackle the Full Software Development Lifecycle: A Case Study [72.24266814625685]
  We explore the performance of large language models (LLMs) across the entire software development lifecycle with DevEval.<n>DevEval features four programming languages, multiple domains, high-quality data collection, and carefully designed and verified metrics for each task.<n> Empirical studies show that current LLMs, including GPT-4, fail to solve the challenges presented within DevEval.
  arXiv  Detail & Related papers  (2024-03-13T15:13:44Z)
• Software Testing with Large Language Models: Survey, Landscape, and Vision [32.34617250991638]
  Pre-trained large language models (LLMs) have emerged as a breakthrough technology in natural language processing and artificial intelligence. This paper provides a comprehensive review of the utilization of LLMs in software testing.
  arXiv  Detail & Related papers  (2023-07-14T08:26:12Z)
• Self-Checker: Plug-and-Play Modules for Fact-Checking with Large Language Models [75.75038268227554]
  Self-Checker is a framework comprising a set of plug-and-play modules that facilitate fact-checking. This framework provides a fast and efficient way to construct fact-checking systems in low-resource environments.
  arXiv  Detail & Related papers  (2023-05-24T01:46:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.