Expectations vs Reality -- A Secondary Study on AI Adoption in Software Testing

• URL: http://arxiv.org/abs/2504.04921v1
• Date: Mon, 07 Apr 2025 11:03:54 GMT
• Title: Expectations vs Reality -- A Secondary Study on AI Adoption in Software Testing
• Authors: Katja Karhu, Jussi Kasurinen, Kari Smolander,
• Abstract summary: In the software industry, artificial intelligence (AI) has been utilized more and more in software development activities.<n>In this paper, the objective was to identify what kind of empirical research with industry context has been conducted on AI in software testing.
• Score: 0.7812210699650152
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: In the software industry, artificial intelligence (AI) has been utilized more and more in software development activities. In some activities, such as coding, AI has already been an everyday tool, but in software testing activities AI it has not yet made a significant breakthrough. In this paper, the objective was to identify what kind of empirical research with industry context has been conducted on AI in software testing, as well as how AI has been adopted in software testing practice. To achieve this, we performed a systematic mapping study of recent (2020 and later) studies on AI adoption in software testing in the industry, and applied thematic analysis to identify common themes and categories, such as the real-world use cases and benefits, in the found papers. The observations suggest that AI is not yet heavily utilized in software testing, and still relatively few studies on AI adoption in software testing have been conducted in the industry context to solve real-world problems. Earlier studies indicated there was a noticeable gap between the actual use cases and actual benefits versus the expectations, which we analyzed further. While there were numerous potential use cases for AI in software testing, such as test case generation, code analysis, and intelligent test automation, the reported actual implementations and observed benefits were limited. In addition, the systematic mapping study revealed a potential problem with false positive search results in online databases when using the search string "artificial intelligence".

Related papers

• Towards an Understanding of Context Utilization in Code Intelligence [37.85380387094615]
  Code intelligence aims to improve the effectiveness and efficiency of various code-related tasks. Recent research suggests that incorporating contextual information beyond the basic original task inputs can substantially enhance model performance. Despite growing academic interest, there is a lack of systematic analysis of context in code intelligence.
  arXiv  Detail & Related papers  (2025-04-11T17:59:53Z)
• Automated Soap Opera Testing Directed by LLMs and Scenario Knowledge: Feasibility, Challenges, and Road Ahead [43.15092098658384]
  Exploratory testing (ET) harnesses tester's knowledge, creativity, and experience to create varying tests that uncover unexpected bugs from the end-user's perspective. We explore the feasibility, challenges and road ahead of automated scenario-based ET (a.k.a soap opera testing)
  arXiv  Detail & Related papers  (2024-12-11T17:57:23Z)
• The Role of Artificial Intelligence and Machine Learning in Software Testing [0.14896196009851972]
  Artificial Intelligence (AI) and Machine Learning (ML) have significantly impacted various industries. Software testing, a crucial part of the software development lifecycle (SDLC), ensures the quality and reliability of software products. This paper explores the role of AI and ML in software testing by reviewing existing literature, analyzing current tools and techniques, and presenting case studies.
  arXiv  Detail & Related papers  (2024-09-04T13:25:13Z)
• AI-powered software testing tools: A systematic review and empirical assessment of their features and limitations [1.0344642971058589]
  AI-driven test automation tools show strong potential in improving software quality and reducing manual testing effort. Future research should focus on advancing AI models to improve adaptability, reliability, and robustness in software testing.
  arXiv  Detail & Related papers  (2024-08-31T10:10:45Z)
• A Multi-Year Grey Literature Review on AI-assisted Test Automation [46.97326049485643]
  Test Automation (TA) techniques are crucial for quality assurance in software engineering but face limitations.<n>Given the prevalent usage of AI in industry, sources of truth are held in grey literature as well as the minds of professionals.<n>This study surveys grey literature to explore how AI is adopted in TA, focusing on the problems it solves, its solutions, and the available tools.
  arXiv  Detail & Related papers  (2024-08-12T15:26:36Z)
• Using Machine Learning To Identify Software Weaknesses From Software Requirement Specifications [49.1574468325115]
  This research focuses on finding an efficient machine learning algorithm to identify software weaknesses from requirement specifications. Keywords extracted using latent semantic analysis help map the CWE categories to PROMISE_exp. Naive Bayes, support vector machine (SVM), decision trees, neural network, and convolutional neural network (CNN) algorithms were tested.
  arXiv  Detail & Related papers  (2023-08-10T13:19:10Z)
• Comparing Software Developers with ChatGPT: An Empirical Investigation [0.0]
  This paper conducts an empirical investigation, contrasting the performance of software engineers and AI systems, like ChatGPT, across different evaluation metrics. The paper posits that a comprehensive comparison of software engineers and AI-based solutions, considering various evaluation criteria, is pivotal in fostering human-machine collaboration.
  arXiv  Detail & Related papers  (2023-05-19T17:25:54Z)
• Metaethical Perspectives on 'Benchmarking' AI Ethics [81.65697003067841]
  Benchmarks are seen as the cornerstone for measuring technical progress in Artificial Intelligence (AI) research. An increasingly prominent research area in AI is ethics, which currently has no set of benchmarks nor commonly accepted way for measuring the 'ethicality' of an AI system. We argue that it makes more sense to talk about 'values' rather than 'ethics' when considering the possible actions of present and future AI systems.
  arXiv  Detail & Related papers  (2022-04-11T14:36:39Z)
• Enabling Automated Machine Learning for Model-Driven AI Engineering [60.09869520679979]
  We propose a novel approach to enable Model-Driven Software Engineering and Model-Driven AI Engineering. In particular, we support Automated ML, thus assisting software engineers without deep AI knowledge in developing AI-intensive systems.
  arXiv  Detail & Related papers  (2022-03-06T10:12:56Z)
• Artificial Intelligence in Software Testing : Impact, Problems, Challenges and Prospect [0.0]
  The study aims to recognize and explain some of the biggest challenges software testers face while applying AI to testing. The paper also proposes some key contributions of AI in the future to the domain of software testing.
  arXiv  Detail & Related papers  (2022-01-14T10:21:51Z)
• Quality Management of Machine Learning Systems [0.0]
  Artificial Intelligence (AI) has become a part of our daily lives due to major advances in Machine Learning (ML) techniques. For business/mission-critical systems, serious concerns about reliability and maintainability of AI applications remain. This paper presents a view of a holistic quality management framework for ML applications based on the current advances.
  arXiv  Detail & Related papers  (2020-06-16T21:34:44Z)
• Machine Learning for Software Engineering: A Systematic Mapping [73.30245214374027]
  The software development industry is rapidly adopting machine learning for transitioning modern day software systems towards highly intelligent and self-learning systems. No comprehensive study exists that explores the current state-of-the-art on the adoption of machine learning across software engineering life cycle stages. This study introduces a machine learning for software engineering (MLSE) taxonomy classifying the state-of-the-art machine learning techniques according to their applicability to various software engineering life cycle stages.
  arXiv  Detail & Related papers  (2020-05-27T11:56:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.