Works on My QPU: Reproducibility in Quantum Computing Research

• URL: http://arxiv.org/abs/2607.08348v1
• Date: Thu, 09 Jul 2026 10:53:02 GMT
• Title: Works on My QPU: Reproducibility in Quantum Computing Research
• Abstract summary: We present a combined manual and automated large-scale analysis of the landscape in quantum computing research.<n>We manually evaluate a curated sample of 127 papers using a five-question framework that covers code availability, environment specification, documentation, hardware description, and executability.<n>Our manual analysis reveals that only 24.4% of the sampled papers provide code artefacts, and among those, 64.5% fail to execute successfully in a clean environment.
• Score: 1.0482211230201115
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Quantum computing research increasingly depends on complex software stacks, yet the reproducibility of published results does not receive the priority and longevity mandated by recommendations of large international scientific bodies and best practices in software-centric systems research. In this paper, we present a combined manual and automated large-scale analysis of the reproducibility landscape in quantum computing research, quantify shortcomings, and derive actionable steps forward. We manually evaluate a curated sample of 127 papers using a five-question framework that covers code availability, environment specification, documentation, hardware description, and executability. To place these findings in a broader context, we conduct an automated large-scale screening of nearly 5000 quantum computing papers for the same reproducibility indicators. Our manual analysis reveals that only 24.4% of the sampled papers provide code artefacts, and among those, 64.5% fail to execute successfully in a clean environment. This assessment is corroborated by a large-scale automated analysis that yields a consistent code availability rate of 26.8%. Further, it shows that approximately one-third of the papers with accessible code lack machine-readable environment specifications. The results in this paper indicate that reproducibility is not yet consistently achieved in quantum computing research. In response, we outline a set of practical recommendations that address the observed failure modes and illustrate how reproducibility can be improved in practice.

Related papers

• AI for Auto-Research: Roadmap & User Guide [107.0834449839233]
  We present an end-to-end analysis of AI across the complete research lifecycle.<n>We identify a sharp, stage-dependent boundary between reliable assistance and unreliable autonomy.<n>We show that greater automation can obscure rather than eliminate failure modes.
  arXiv  Detail & Related papers  (2026-05-18T17:08:26Z)
• MLReplicate: Benchmarking Autonomous Research Systems for Machine Learning Reproducibility [2.2969445973129123]
  We introduce MLReplicate, an end-to-end benchmark evaluating autonomous research systems on machine learning.<n>The benchmark was constructed from ICML 2025 papers reformulated into standardized input specifications and evaluated across 6 state-of-the-art research systems.<n>In contrast to automated reviews, human reviewers consistently identified methodological flaws, hallucinated experimental results, and failures across all systems.
  arXiv  Detail & Related papers  (2026-05-15T20:35:32Z)
• Can Coding Agents Reproduce Findings in Computational Materials Science? [49.254975563645786]
  We present AutoMat, a benchmark for evaluating large language models' ability to reproduce scientific claims.<n>By working closely with subject matter experts, we curate a set of claims from real materials science papers to test whether coding agents can recover and execute the end-to-end workflow.<n>Results show that current LLM-based agents obtain low overall success rates on AutoMat, with the best-performing setting achieving a success rate of only 54.1%.
  arXiv  Detail & Related papers  (2026-05-01T17:42:12Z)
• PRBench: End-to-end Paper Reproduction in Physics Research [32.672534450424386]
  PRBench is a benchmark of 30 expert-curated tasks spanning 11 subfields of physics.<n>Agents are provided only with the task instruction and paper content, and operate in a sandboxed execution environment.<n>We evaluate a set of coding agents on PRBench and analyze their capabilities across key dimensions of scientific reasoning and execution.
  arXiv  Detail & Related papers  (2026-03-29T11:44:57Z)
• Assessing Reproducibility in Evolutionary Computation: A Case Study using Human- and LLM-based Assessment [2.0365636651755263]
  We study the practices in papers published in the Combinatorial Optimization and Metaheuristics track of the Evolutionary Computation Conference over a ten-year period.<n>We introduce a structured checklist and apply it through a systematic manual assessment of the selected corpus.<n>In addition, we propose RECAP (REproducibility Checklist Automation Pipeline), an automated system that automatically evaluates signals from paper text and associated code.
  arXiv  Detail & Related papers  (2026-02-05T08:32:29Z)
• SciCoQA: Quality Assurance for Scientific Paper--Code Alignment [53.70401063640645]
  We present SciCoQA, a dataset for detecting discrepancies between scientific publications and theirs.<n>Our dataset consists of 611 paper-code discrepancies (81 real, 530 synthetic), spanning diverse computational science disciplines.<n>The best performing model in our evaluation, GPT-5, can only detect 45.7% of real-world paper-code discrepancies.
  arXiv  Detail & Related papers  (2026-01-19T10:04:33Z)
• RPC-Bench: A Fine-grained Benchmark for Research Paper Comprehension [65.81339691942757]
  RPC-Bench is a large-scale question-answering benchmark built from review-rebuttal exchanges of high-quality computer science papers.<n>We design a fine-grained taxonomy aligned with the scientific research flow to assess models' ability to understand and answer why, what, and how questions in scholarly contexts.
  arXiv  Detail & Related papers  (2026-01-14T11:37:00Z)
• AutoMalDesc: Large-Scale Script Analysis for Cyber Threat Research [81.04845910798387]
  Generating natural language explanations for threat detections remains an open problem in cybersecurity research.<n>We present AutoMalDesc, an automated static analysis summarization framework that operates independently at scale.<n>We publish our complete dataset of more than 100K script samples, including annotated seed (0.9K) datasets, along with our methodology and evaluation framework.
  arXiv  Detail & Related papers  (2025-11-17T13:05:25Z)
• Computational Reproducibility of R Code Supplements on OSF [0.0]
  Many published code supplements lack the necessary documentation to recreate their computational environments.<n>In this work, we assess the computational of 296 R projects using the StatCodeSearch dataset.<n>We developed an automated pipeline that reconstructs computational environments directly from project source code.
  arXiv  Detail & Related papers  (2025-05-27T13:06:38Z)
• AutoReproduce: Automatic AI Experiment Reproduction with Paper Lineage [62.049868205196425]
  AutoReproduce is a framework capable of automatically reproducing experiments described in research papers in an end-to-end manner.<n>Results show that AutoReproduce achieves an average performance gap of $22.1%$ on $89.74%$ of the executable experiment runs.
  arXiv  Detail & Related papers  (2025-05-27T03:15:21Z)
• Investigating Reproducibility in Deep Learning-Based Software Fault Prediction [16.25827159504845]
  With the rapid adoption of increasingly complex machine learning models, it becomes more and more difficult for scholars to reproduce the results that are reported in the literature. This is in particular the case when the applied deep learning models and the evaluation methodology are not properly documented and when code and data are not shared. We have conducted a systematic review of the current literature and examined the level of 56 research articles that were published between 2019 and 2022 in top-tier software engineering conferences.
  arXiv  Detail & Related papers  (2024-02-08T13:00:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.