QuCheck: A Property-based Testing Framework for Quantum Programs in Qiskit

• URL: http://arxiv.org/abs/2503.22641v1
• Date: Fri, 28 Mar 2025 17:30:09 GMT
• Title: QuCheck: A Property-based Testing Framework for Quantum Programs in Qiskit
• Abstract summary: Property-based testing has been previously proposed for quantum programs in Q# with QSharpCheck.<n>We propose QuCheck, an enhanced property-based testing framework in Qiskit.
• Score: 0.5735035463793009
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Property-based testing has been previously proposed for quantum programs in Q# with QSharpCheck; however, this implementation was limited in functionality, lacked extensibility, and was evaluated on a narrow range of programs using a single property. To address these limitations, we propose QuCheck, an enhanced property-based testing framework in Qiskit. By leveraging Qiskit and the broader Python ecosystem, QuCheck facilitates property construction, introduces flexible input generators and assertions, and supports expressive preconditions. We assessed its effectiveness through mutation analysis on five quantum programs (2-10 qubits), varying the number of properties, inputs, and measurement shots to assess their impact on fault detection and demonstrate the effectiveness of property-based testing across a range of conditions. Results show a strong positive correlation between the mutation score (a measure of fault detection) and number of properties evaluated, with a moderate negative correlation between the false positive rate and number of measurement shots. Among the most thorough test configurations, those evaluating three properties achieved a mean mutation score ranging from 0.90 to 0.92 across all five algorithms, with the false positive rate between 0 and 0.04. QuCheck identified 36.0% more faults than QSharpCheck, with execution time reduced by 81.1%, despite one false positive. These findings underscore the viability of property-based testing for verifying quantum systems.

Related papers

• Auditing and Decomposing Feedback-Driven Evolution in LLM Test Generation under the Oracle Problem [14.44743150931186]
  invalid or underspecified inputs can create spurious fault detections and apparent evolutionary gains.<n>We audit this failure mode in feedback-driven test generation using 142 development tasks, 114 locked external tasks, and 138 held-out tasks.<n>We propose an audit-and-placebo protocol that separates verifier artifacts, interaction scaffolding, and grounded feedback credit in evaluations of self-evolving test generators.
  arXiv  Detail & Related papers  (2026-08-20T04:29:35Z)
• TestifAI: Tomography-Based Testing for Deep Learning Systems [42.86570387250456]
  TestifAI is a framework for efficient and accurate estimation of robustness against combinations of perturbations.<n>It reconstructs model behaviour in a multi-perturbation space from tests that apply only a small number of perturbations.<n>It can predict higher-order (3 and 4 perturbations) test outcomes from low-order (1 and 2 perturbations) observations with an aggregate robustness estimation error of less than 7%.
  arXiv  Detail & Related papers  (2026-08-19T13:25:50Z)
• CodeAssay: A Multi-Metric Benchmark with Audited Ground Truth for LLM Code Generation [1.8589311604031529]
  We present CodeAssay, a taxonomy-first benchmark of 185 Python tasks across ten software-engineering categories.<n>It combines audited ground truth, public tests for generation and repair, hidden tests for grading, mutation-based test-suite validation, and selected code-property measures.
  arXiv  Detail & Related papers  (2026-08-04T12:15:17Z)
• GroverFigureOfMerit: An Agnostic Figure of Merit for Quantum Backend Characterization in the NISQ Era [41.99844472131922]
  We propose a Figure of Merit based on Grover's algorithm as an algorithmic stress test evaluating quantum backend performance holistically.<n>The metric combines success probability on target states with penalties for non-uniform amplification and leakage to non-marked states, yielding a unified score across hardware architectures.
  arXiv  Detail & Related papers  (2026-07-09T16:08:48Z)
• Agentic Proving for Program Verification [44.663012714194025]
  Agentic systems have emerged as state-of-the-art approaches for automated theorem proving in formal mathematics.<n>We evaluate Claude Code in an agentic proving framework on CLEVER, a Lean 4 benchmark for verifiable code generation.
  arXiv  Detail & Related papers  (2026-05-22T15:41:27Z)
• Are Benchmark Tests Strong Enough? Mutation-Guided Diagnosis and Augmentation of Regression Suites [49.16055123488827]
  Insufficiently strong test suites can admit plausible yet semantically incorrect patches, inflating reported success rates.<n>We introduce STING, a framework for targeted test augmentation that uses semantically altered program variants as diagnostic stressors.<n>STING produces 1,014 validated tests spanning 211 instances and increases patch-region line and branch coverage by 10.8% and 9.5%.
  arXiv  Detail & Related papers  (2026-04-02T01:13:40Z)
• Detecting Flaky Tests in Quantum Software: A Dynamic Approach [4.46640294257026]
  Flaky tests that pass or fail nondeterministically without changes to code or environment pose a serious threat to software reliability.<n>This paper presents the first large-scale dynamic characterization of flaky tests in quantum software.<n>We executed the Qiskit Terra test suite 10,000 times across 23 releases in controlled environments.
  arXiv  Detail & Related papers  (2025-12-19T21:47:31Z)
• QSentry: Backdoor Detection for Quantum Neural Networks via Measurement Clustering [43.44248599606903]
  Quantum neural networks (QNNs) are an important model for implementing quantum machine learning (QML)<n>This work establishes a practical and effective framework for mitigating backdoor threats in QML.
  arXiv  Detail & Related papers  (2025-11-19T12:08:11Z)
• ORFuzz: Fuzzing the "Other Side" of LLM Safety -- Testing Over-Refusal [27.26251627767238]
  Large Language Models (LLMs) increasingly exhibit over-refusal - erroneously rejecting benign queries due to overly conservative safety measures.<n>This paper introduces the first evolutionary testing framework, ORFuzz, for the systematic detection and analysis of LLM over-refusals.
  arXiv  Detail & Related papers  (2025-08-15T05:03:26Z)
• Intention-Driven Generation of Project-Specific Test Cases [45.2380093475221]
  We propose IntentionTest, which generates project-specific tests given the description of validation intention.<n>We extensively evaluate IntentionTest against state-of-the-art baselines (DA, ChatTester, and EvoSuite) on 4,146 test cases from 13 open-source projects.
  arXiv  Detail & Related papers  (2025-07-28T08:35:04Z)
• Calibration of Quantum Devices via Robust Statistical Methods [45.464983015777314]
  We numerically analyze advanced statistical methods for Bayesian inference against the state-of-the-art in quantum parameter learning.<n>We show advantages of these approaches over existing ones, namely under multi-modality and high dimensionality.<n>Our findings have applications in challenging quantumcharacterization tasks namely learning the dynamics of open quantum systems.
  arXiv  Detail & Related papers  (2025-07-09T15:22:17Z)
• DS@GT at CheckThat! 2025: Evaluating Context and Tokenization Strategies for Numerical Fact Verification [49.1574468325115]
  Numerical claims, statements involving quantities, comparisons, and temporal references pose unique challenges for automated fact-checking systems.<n>We evaluate modeling strategies for veracity prediction of such claims using the QuanTemp dataset and building our own evidence retrieval pipeline.<n>Our best-performing system achieves competitive macro-average F1 score of 0.57 and places us among the Top-4 submissions in Task 3 of CheckThat! 2025.
  arXiv  Detail & Related papers  (2025-07-08T17:22:22Z)
• Bloch Vector Assertions for Debugging Quantum Programs [3.8028747063484594]
  Bloq is a scalable, automated fault localization approach.<n>We introduce AutoBloq, a component of Bloq for automatically generating assertion schemes from quantum algorithms.
  arXiv  Detail & Related papers  (2025-06-23T09:53:02Z)
• Bug Classification in Quantum Software: A Rule-Based Framework and Its Evaluation [1.1510009152620668]
  This paper presents an automated framework for classifying issues in quantum software repositories by bug type, category, severity, and impacted quality attributes.<n>The framework achieves up to 85.21% accuracy, with F1-scores ranging from 0.7075 (severity) to 0.8393 (quality attribute)<n>A review of 1,550 quantum-specific bugs showed that over half involved quantum circuit-level problems, followed by gate errors and hardware-related issues.
  arXiv  Detail & Related papers  (2025-06-12T06:42:10Z)
• VALTEST: Automated Validation of Language Model Generated Test Cases [0.7059472280274008]
  Large Language Models (LLMs) have demonstrated significant potential in automating software testing, specifically in generating unit test cases. This paper introduces VALTEST, a novel framework designed to automatically validate test cases generated by LLMs by leveraging token probabilities.
  arXiv  Detail & Related papers  (2024-11-13T00:07:32Z)
• FactTest: Factuality Testing in Large Language Models with Finite-Sample and Distribution-Free Guarantees [41.78390564658645]
  Large Language Models (LLMs) to generate hallucinations and non-factual content undermines their reliability in high-stakes domains. We introduce FactTest, a novel framework that statistically assesses whether a LLM can confidently provide correct answers to given questions. We show that FactTest effectively detects hallucinations and improves the model's ability to abstain from answering unknown questions, leading to an over 40% accuracy improvement.
  arXiv  Detail & Related papers  (2024-11-04T20:53:04Z)
• Automating Quantum Software Maintenance: Flakiness Detection and Root Cause Analysis [4.554856650068748]
  Flaky tests, which pass or fail inconsistently without code changes, are a major challenge in software engineering. We aim to create an automated framework to detect flaky tests in quantum software.
  arXiv  Detail & Related papers  (2024-10-31T02:43:04Z)
• QuanTest: Entanglement-Guided Testing of Quantum Neural Network Systems [45.18451374144537]
  Quantum Neural Network (QNN) combines the Deep Learning (DL) principle with the fundamental theory of quantum mechanics to achieve machine learning tasks with quantum acceleration. QNN systems differ significantly from traditional quantum software and classical DL systems, posing critical challenges for QNN testing. We propose QuanTest, a quantum entanglement-guided adversarial testing framework to uncover potential erroneous behaviors in QNN systems.
  arXiv  Detail & Related papers  (2024-02-20T12:11:28Z)
• Calibrated Uncertainty Quantification for Operator Learning via Conformal Prediction [95.75771195913046]
  We propose a risk-controlling quantile neural operator, a distribution-free, finite-sample functional calibration conformal prediction method. We provide a theoretical calibration guarantee on the coverage rate, defined as the expected percentage of points on the function domain. Empirical results on a 2D Darcy flow and a 3D car surface pressure prediction task validate our theoretical results.
  arXiv  Detail & Related papers  (2024-02-02T23:43:28Z)
• Precise Error Rates for Computationally Efficient Testing [75.63895690909241]
  We revisit the question of simple-versus-simple hypothesis testing with an eye towards computational complexity. An existing test based on linear spectral statistics achieves the best possible tradeoff curve between type I and type II error rates.
  arXiv  Detail & Related papers  (2023-11-01T04:41:16Z)
• Contextual Predictive Mutation Testing [17.832774161583036]
  We introduce MutationBERT, an approach for predictive mutation testing that simultaneously encodes the source method mutation and test method. Thanks to its higher precision, MutationBERT saves 33% of the time spent by a prior approach on checking/verifying live mutants. We validate our input representation, and aggregation approaches for lifting predictions from the test matrix level to the test suite level, finding similar improvements in performance.
  arXiv  Detail & Related papers  (2023-09-05T17:00:15Z)
• Effective Test Generation Using Pre-trained Large Language Models and Mutation Testing [13.743062498008555]
  We introduce MuTAP for improving the effectiveness of test cases generated by Large Language Models (LLMs) in terms of revealing bugs. MuTAP is capable of generating effective test cases in the absence of natural language descriptions of the Program Under Test (PUTs) Our results show that our proposed method is able to detect up to 28% more faulty human-written code snippets.
  arXiv  Detail & Related papers  (2023-08-31T08:48:31Z)
• From Static Benchmarks to Adaptive Testing: Psychometrics in AI Evaluation [60.14902811624433]
  We discuss a paradigm shift from static evaluation methods to adaptive testing. This involves estimating the characteristics and value of each test item in the benchmark and dynamically adjusting items in real-time. We analyze the current approaches, advantages, and underlying reasons for adopting psychometrics in AI evaluation.
  arXiv  Detail & Related papers  (2023-06-18T09:54:33Z)
• An ensemble meta-estimator to predict source code testability [1.4213973379473652]
  The size of a test suite determines the test effort and cost, while the coverage measure indicates the test effectiveness. This paper offers a new equation to estimate testability regarding the size and coverage of a given test suite.
  arXiv  Detail & Related papers  (2022-08-20T06:18:16Z)
• Statistical and Computational Phase Transitions in Group Testing [73.55361918807883]
  We study the group testing problem where the goal is to identify a set of k infected individuals carrying a rare disease. We consider two different simple random procedures for assigning individuals tests.
  arXiv  Detail & Related papers  (2022-06-15T16:38:50Z)
• Measuring NISQ Gate-Based Qubit Stability Using a 1+1 Field Theory and Cycle Benchmarking [50.8020641352841]
  We study coherent errors on a quantum hardware platform using a transverse field Ising model Hamiltonian as a sample user application. We identify inter-day and intra-day qubit calibration drift and the impacts of quantum circuit placement on groups of qubits in different physical locations on the processor. This paper also discusses how these measurements can provide a better understanding of these types of errors and how they may improve efforts to validate the accuracy of quantum computations.
  arXiv  Detail & Related papers  (2022-01-08T23:12:55Z)
• Deconfounding Scores: Feature Representations for Causal Effect Estimation with Weak Overlap [140.98628848491146]
  We introduce deconfounding scores, which induce better overlap without biasing the target of estimation. We show that deconfounding scores satisfy a zero-covariance condition that is identifiable in observed data. In particular, we show that this technique could be an attractive alternative to standard regularizations.
  arXiv  Detail & Related papers  (2021-04-12T18:50:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.