Automating Detection and Root-Cause Analysis of Flaky Tests in Quantum Software

• URL: http://arxiv.org/abs/2603.09029v1
• Date: Mon, 09 Mar 2026 23:57:55 GMT
• Title: Automating Detection and Root-Cause Analysis of Flaky Tests in Quantum Software
• Abstract summary: This paper presents an automated pipeline to detect flaky-test-related issues and pull requests in quantum software repositories.<n>We expand an existing quantum flaky test dataset and evaluate the capability of Large Language Models for flakiness classification and root-cause identification.<n>The best-performing model, Google Gemini, achieves an F1-score of 0.9420 for flakiness detection and 0.9643 for root-cause identification.
• Score: 3.853925623717688
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Like classical software, quantum software systems rely on automated testing. However, their inherently probabilistic outputs make them susceptible to quantum flakiness -- tests that pass or fail inconsistently without code changes. Such quantum flaky tests can mask real defects and reduce developer productivity, yet systematic tooling for their detection and diagnosis remains limited. This paper presents an automated pipeline to detect flaky-test-related issues and pull requests in quantum software repositories and to support the identification of their root causes. We aim to expand an existing quantum flaky test dataset and evaluate the capability of Large Language Models (LLMs) for flakiness classification and root-cause identification. Building on a prior manual analysis of 14 quantum software repositories, we automate the discovery of additional flaky test cases using LLMs and cosine similarity. We further evaluate a variety of LLMs from OpenAI GPT, Meta LLaMA, Google Gemini, and Anthropic Claude suites for classifying flakiness and identifying root causes from issue descriptions and code context. Classification performance is assessed using standard performance metrics, including F1-score. Using our pipeline, we identify 25 previously unknown flaky tests, increasing the original dataset size by 54%. The best-performing model, Google Gemini, achieves an F1-score of 0.9420 for flakiness detection and 0.9643 for root-cause identification, demonstrating that LLMs can provide practical support for triaging flaky reports and understanding their underlying causes in quantum software. The expanded dataset and automated pipeline provide reusable artifacts for the quantum software engineering community. Future work will focus on improving detection robustness and exploring automated repair of quantum flaky tests.

Related papers

• GroverFigureOfMerit: An Agnostic Figure of Merit for Quantum Backend Characterization in the NISQ Era [41.99844472131922]
  We propose a Figure of Merit based on Grover's algorithm as an algorithmic stress test evaluating quantum backend performance holistically.<n>The metric combines success probability on target states with penalties for non-uniform amplification and leakage to non-marked states, yielding a unified score across hardware architectures.
  arXiv  Detail & Related papers  (2026-07-09T16:08:48Z)
• LiveFMBench: Unveiling the Power and Limits of Agentic Workflows in Specification Generation [75.05397479715576]
  Large language models (LLMs) and agents have shown promising progress, but their true capabilities and failure modes remain unclear.<n>We present the first systematic and contamination-aware study of LLM- and agent-based formal specification generation for C programs.
  arXiv  Detail & Related papers  (2026-05-02T11:31:33Z)
• VIBEPASS: Can Vibe Coders Really Pass the Vibe Check? [46.85901599242161]
  We evaluate two coupled tasks: emphFault-Triggering Test Generation (FT-Test) and emphFault-targeted Program Repair (FPR)<n>We find that fault-targeted reasoning does not scale with general coding ability.<n>Test-guided repair reveals a complementary insight: when self-generated tests successfully witness a fault, the resulting repair matches or outperforms repair guided by externally provided tests, but tests that fail to witness the fault actively degrade repair below unguided baselines.
  arXiv  Detail & Related papers  (2026-03-16T21:14:28Z)
• DeepFix: Debugging and Fixing Machine Learning Workflow using Agentic AI [0.0]
  DeepFix is a tool for automated testing of the entire machine learning pipeline using an agentic AI framework.<n>Our testing approach first leverages Deepchecks to test the ML software for any potential bugs, and thereafter, uses an agentic AI-based approach to generate a detailed bug report.<n>This includes a ranking, based on the severity of the found bugs, along with their explanations, which can be interpreted easily by any non-data science experts.
  arXiv  Detail & Related papers  (2026-03-14T20:00:16Z)
• Automated Self-Testing as a Quality Gate: Evidence-Driven Release Management for LLM Applications [51.56484100374058]
  We present an automated self-testing framework that introduces quality gates with evidence-based release decisions.<n>We evaluate the framework through a longitudinal case study of an internally deployed multi-agent conversational AI system.
  arXiv  Detail & Related papers  (2026-03-13T20:44:15Z)
• Detecting Flaky Tests in Quantum Software: A Dynamic Approach [4.46640294257026]
  Flaky tests that pass or fail nondeterministically without changes to code or environment pose a serious threat to software reliability.<n>This paper presents the first large-scale dynamic characterization of flaky tests in quantum software.<n>We executed the Qiskit Terra test suite 10,000 times across 23 releases in controlled environments.
  arXiv  Detail & Related papers  (2025-12-19T21:47:31Z)
• Identifying Flaky Tests in Quantum Code: A Machine Learning Approach [5.323578182914324]
  Indeterminacy, a fundamental characteristic of quantum systems, increases the likelihood of flaky tests in quantum programs.<n>We present a novel machine learning platform that leverages multiple machine learning models to automatically detect flaky tests in quantum programs.
  arXiv  Detail & Related papers  (2025-02-06T19:43:51Z)
• Automating Quantum Software Maintenance: Flakiness Detection and Root Cause Analysis [4.554856650068748]
  Flaky tests, which pass or fail inconsistently without code changes, are a major challenge in software engineering. We aim to create an automated framework to detect flaky tests in quantum software.
  arXiv  Detail & Related papers  (2024-10-31T02:43:04Z)
• Quantum Patch-Based Autoencoder for Anomaly Segmentation [44.99833362998488]
  We introduce a patch-based quantum autoencoder (QPB-AE) for image anomaly segmentation. QPB-AE reconstructs the quantum state of the embedded input patches, computing an anomaly map directly from measurement. We evaluate its performance across multiple datasets and parameter configurations.
  arXiv  Detail & Related papers  (2024-04-26T08:42:58Z)
• QuanTest: Entanglement-Guided Testing of Quantum Neural Network Systems [45.18451374144537]
  Quantum Neural Network (QNN) combines the Deep Learning (DL) principle with the fundamental theory of quantum mechanics to achieve machine learning tasks with quantum acceleration. QNN systems differ significantly from traditional quantum software and classical DL systems, posing critical challenges for QNN testing. We propose QuanTest, a quantum entanglement-guided adversarial testing framework to uncover potential erroneous behaviors in QNN systems.
  arXiv  Detail & Related papers  (2024-02-20T12:11:28Z)
• QuCAT: A Combinatorial Testing Tool for Quantum Software [43.8945621250487]
  We present the quantum software testing tool QuCAT for testing of quantum programs. With the first functionality, the tool generates a test suite of a given strength. With the second functionality, it generates test suites with increasing strength until a failure is triggered.
  arXiv  Detail & Related papers  (2023-08-31T20:17:38Z)
• Identifying Flakiness in Quantum Programs [5.592360872268223]
  We find flaky tests in 12 out of 14 quantum software repositories. We identify 46 distinct flaky test reports with 8 groups of causes and 7 common solutions. This work may interest practitioners, as it provides useful insight into the resolution of flaky tests in quantum programs.
  arXiv  Detail & Related papers  (2023-02-07T04:55:34Z)
• Validation tests of GBS quantum computers give evidence for quantum advantage with a decoherent target [62.997667081978825]
  We use positive-P phase-space simulations of grouped count probabilities as a fingerprint for verifying multi-mode data. We show how one can disprove faked data, and apply this to a classical count algorithm.
  arXiv  Detail & Related papers  (2022-11-07T12:00:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.