Automating Quantum Software Maintenance: Flakiness Detection and Root Cause Analysis

• URL: http://arxiv.org/abs/2410.23578v1
• Date: Thu, 31 Oct 2024 02:43:04 GMT
• Title: Automating Quantum Software Maintenance: Flakiness Detection and Root Cause Analysis
• Authors: Janakan Sivaloganathan, Ainaz Jamshidi, Andriy Miranskyy, Lei Zhang,
• Abstract summary: Flaky tests, which pass or fail inconsistently without code changes, are a major challenge in software engineering. We aim to create an automated framework to detect flaky tests in quantum software.
• Score: 4.554856650068748
• License:
• Abstract: Flaky tests, which pass or fail inconsistently without code changes, are a major challenge in software engineering in general and in quantum software engineering in particular due to their complexity and probabilistic nature, leading to hidden issues and wasted developer effort. We aim to create an automated framework to detect flaky tests in quantum software and an extended dataset of quantum flaky tests, overcoming the limitations of manual methods. Building on prior manual analysis of 14 quantum software repositories, we expanded the dataset and automated flaky test detection using transformers and cosine similarity. We conducted experiments with Large Language Models (LLMs) from the OpenAI GPT and Meta LLaMA families to assess their ability to detect and classify flaky tests from code and issue descriptions. Embedding transformers proved effective: we identified 25 new flaky tests, expanding the dataset by 54%. Top LLMs achieved an F1-score of 0.8871 for flakiness detection but only 0.5839 for root cause identification. We introduced an automated flaky test detection framework using machine learning, showing promising results but highlighting the need for improved root cause detection and classification in large quantum codebases. Future work will focus on improving detection techniques and developing automatic flaky test fixes.

Related papers

• AutoPT: How Far Are We from the End2End Automated Web Penetration Testing? [54.65079443902714]
  We introduce AutoPT, an automated penetration testing agent based on the principle of PSM driven by LLMs. Our results show that AutoPT outperforms the baseline framework ReAct on the GPT-4o mini model.
  arXiv  Detail & Related papers  (2024-11-02T13:24:30Z)
• FuzzTheREST: An Intelligent Automated Black-box RESTful API Fuzzer [0.0]
  This work introduces a black-box API of fuzzy testing tool that employs Reinforcement Learning (RL) for vulnerability detection. The tool found a total of six unique vulnerabilities and achieved 55% code coverage.
  arXiv  Detail & Related papers  (2024-07-19T14:43:35Z)
• FlaKat: A Machine Learning-Based Categorization Framework for Flaky Tests [3.0846824529023382]
  Flaky tests can pass or fail non-deterministically, without alterations to a software system. State-of-the-art research incorporates machine learning solutions into flaky test detection and achieves reasonably good accuracy.
  arXiv  Detail & Related papers  (2024-03-01T22:00:44Z)
• Identifying Flakiness in Quantum Programs [5.592360872268223]
  We find flaky tests in 12 out of 14 quantum software repositories. We identify 46 distinct flaky test reports with 8 groups of causes and 7 common solutions. This work may interest practitioners, as it provides useful insight into the resolution of flaky tests in quantum programs.
  arXiv  Detail & Related papers  (2023-02-07T04:55:34Z)
• Validation tests of GBS quantum computers give evidence for quantum advantage with a decoherent target [62.997667081978825]
  We use positive-P phase-space simulations of grouped count probabilities as a fingerprint for verifying multi-mode data. We show how one can disprove faked data, and apply this to a classical count algorithm.
  arXiv  Detail & Related papers  (2022-11-07T12:00:45Z)
• Experimental benchmarking of an automated deterministic error suppression workflow for quantum algorithms [0.0]
  Excitement about the promise of quantum computers is tempered by the reality that the hardware remains exceptionally fragile and error-prone. We describe and experimentally test a fully autonomous workflow designed to deterministically suppress errors in quantum algorithms from the gate level through to circuit execution and measurement.
  arXiv  Detail & Related papers  (2022-09-14T18:23:17Z)
• SUPERNOVA: Automating Test Selection and Defect Prevention in AAA Video Games Using Risk Based Testing and Machine Learning [62.997667081978825]
  Testing video games is an increasingly difficult task as traditional methods fail to scale with growing software systems. We present SUPERNOVA, a system responsible for test selection and defect prevention while also functioning as an automation hub. The direct impact of this has been observed to be a reduction in 55% or more testing hours for an undisclosed sports game title.
  arXiv  Detail & Related papers  (2022-03-10T00:47:46Z)
• DAE : Discriminatory Auto-Encoder for multivariate time-series anomaly detection in air transportation [68.8204255655161]
  We propose a novel anomaly detection model called Discriminatory Auto-Encoder (DAE) It uses the baseline of a regular LSTM-based auto-encoder but with several decoders, each getting data of a specific flight phase. Results show that the DAE achieves better results in both accuracy and speed of detection.
  arXiv  Detail & Related papers  (2021-09-08T14:07:55Z)
• ALT-MAS: A Data-Efficient Framework for Active Testing of Machine Learning Algorithms [58.684954492439424]
  We propose a novel framework to efficiently test a machine learning model using only a small amount of labeled test data. The idea is to estimate the metrics of interest for a model-under-test using Bayesian neural network (BNN)
  arXiv  Detail & Related papers  (2021-04-11T12:14:04Z)
• What is the Vocabulary of Flaky Tests? An Extended Replication [0.0]
  We conduct an empirical study to assess the use of code identifiers to predict test flakiness. We validated the performance of trained models using datasets with other flaky tests and from different projects.
  arXiv  Detail & Related papers  (2021-03-23T16:42:22Z)
• Anomaly Detection Based on Selection and Weighting in Latent Space [73.01328671569759]
  We propose a novel selection-and-weighting-based anomaly detection framework called SWAD. Experiments on both benchmark and real-world datasets have shown the effectiveness and superiority of SWAD.
  arXiv  Detail & Related papers  (2021-03-08T10:56:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.