AI-powered test automation tools: A systematic review and empirical evaluation

• URL: http://arxiv.org/abs/2409.00411v1
• Date: Sat, 31 Aug 2024 10:10:45 GMT
• Title: AI-powered test automation tools: A systematic review and empirical evaluation
• Authors: Vahid Garousi, Nithin Joy, Alper Buğra Keleş,
• Abstract summary: We investigate the features provided by existing AI-based test automation tools. We empirically evaluate how the AI features can be helpful for effectiveness and efficiency of testing. We also study the limitations of the AI features in AI-based test tools.
• Score: 1.3490988186255937
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Context: Test engineers are looking at more ways to test systems more effectively and efficiently. With recent advances in the field of AI (Artificial Intelligence), a large number of AI-powered test automation tools have emerged, which can help make testing more effective and efficient. Objective: We investigate the features provided by existing AI-based test automation tools. We also empirically evaluate the performance of two AI-based testing tools by applying them on two selected open-source Software Under Test (SUT). We empirically evaluate how the AI features can be helpful for effectiveness and efficiency of testing. We also study the limitations of the AI features in AI-based test tools. Method: To accomplish the objective, a Multivocal Literature Review (MLR) study was conducted to investigate the landscape of the AI-based test automation tools in the industry. Moreover, an empirical assessment is also conducted to empirically analyze two AI-based test automation tools by using it on two open-source projects. To determine the need of AI for selected feature, the same feature was created without the use of ML to explore its limitations which can be avoided using AI. Results: Our results are based on 55 AI-based test automation tools. Furthermore, an empirical assessment was performed by selecting two of the 55 tools analyzed as part of the MLR. Conclusion: This paper explores the potential benefits and limitations of AI-based test automation tools. The limitations explored can be used as inspiration to develop better AI-based test tools.

Related papers

• AI-Compass: A Comprehensive and Effective Multi-module Testing Tool for AI Systems [26.605694684145313]
  In this study, we design and implement a testing tool, tool, to comprehensively and effectively evaluate AI systems. The tool extensively assesses adversarial robustness, model interpretability, and performs neuron analysis. Our research sheds light on a general solution for AI systems testing landscape.
  arXiv  Detail & Related papers  (2024-11-09T11:15:17Z)
• Disrupting Test Development with AI Assistants [1.024113475677323]
  Generative AI-assisted coding tools like GitHub Copilot, ChatGPT, and Tabnine have significantly transformed software development. This paper analyzes how these innovations impact productivity and software test development metrics.
  arXiv  Detail & Related papers  (2024-11-04T17:52:40Z)
• AutoPT: How Far Are We from the End2End Automated Web Penetration Testing? [54.65079443902714]
  We introduce AutoPT, an automated penetration testing agent based on the principle of PSM driven by LLMs. Our results show that AutoPT outperforms the baseline framework ReAct on the GPT-4o mini model.
  arXiv  Detail & Related papers  (2024-11-02T13:24:30Z)
• Raising the Stakes: Performance Pressure Improves AI-Assisted Decision Making [57.53469908423318]
  We show the effects of performance pressure on AI advice reliance when laypeople complete a common AI-assisted task. We find that when the stakes are high, people use AI advice more appropriately than when stakes are lower, regardless of the presence of an AI explanation.
  arXiv  Detail & Related papers  (2024-10-21T22:39:52Z)
• ExACT: Teaching AI Agents to Explore with Reflective-MCTS and Exploratory Learning [78.42927884000673]
  ExACT is an approach to combine test-time search and self-learning to build o1-like models for agentic applications. We first introduce Reflective Monte Carlo Tree Search (R-MCTS), a novel test time algorithm designed to enhance AI agents' ability to explore decision space on the fly. Next, we introduce Exploratory Learning, a novel learning strategy to teach agents to search at inference time without relying on any external search algorithms.
  arXiv  Detail & Related papers  (2024-10-02T21:42:35Z)
• The Role of Artificial Intelligence and Machine Learning in Software Testing [0.14896196009851972]
  Artificial Intelligence (AI) and Machine Learning (ML) have significantly impacted various industries. Software testing, a crucial part of the software development lifecycle (SDLC), ensures the quality and reliability of software products. This paper explores the role of AI and ML in software testing by reviewing existing literature, analyzing current tools and techniques, and presenting case studies.
  arXiv  Detail & Related papers  (2024-09-04T13:25:13Z)
• A Multi-Year Grey Literature Review on AI-assisted Test Automation [46.97326049485643]
  Test Automation (TA) techniques are crucial for quality assurance in software engineering. TA techniques face limitations such as high test suite maintenance costs and the need for extensive programming skills. Artificial Intelligence (AI) offers new opportunities to address these issues through automation and improved practices.
  arXiv  Detail & Related papers  (2024-08-12T15:26:36Z)
• Towards AI Accountability Infrastructure: Gaps and Opportunities in AI Audit Tooling [1.841662059101602]
  Audits are critical mechanisms for identifying the risks and limitations of deployed artificial intelligence (AI) systems. We map the current ecosystem of available AI audit tools. We conclude that resources are lacking to adequately support the full scope of needs for many AI audit practitioners.
  arXiv  Detail & Related papers  (2024-02-27T19:52:54Z)
• Testing autonomous vehicles and AI: perspectives and challenges from cybersecurity, transparency, robustness and fairness [53.91018508439669]
  The study explores the complexities of integrating Artificial Intelligence into Autonomous Vehicles (AVs) It examines the challenges introduced by AI components and the impact on testing procedures. The paper identifies significant challenges and suggests future directions for research and development of AI in AV technology.
  arXiv  Detail & Related papers  (2024-02-21T08:29:42Z)
• The Role of AI in Drug Discovery: Challenges, Opportunities, and Strategies [97.5153823429076]
  The benefits, challenges and drawbacks of AI in this field are reviewed. The use of data augmentation, explainable AI, and the integration of AI with traditional experimental methods are also discussed.
  arXiv  Detail & Related papers  (2022-12-08T23:23:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.