Metriq: A Collaborative Platform for Benchmarking Quantum Computers

• URL: http://arxiv.org/abs/2603.08680v1
• Date: Mon, 09 Mar 2026 17:49:41 GMT
• Title: Metriq: A Collaborative Platform for Benchmarking Quantum Computers
• Abstract summary: We introduce Metriq, an open-source platform for reproducible cross-platform quantum benchmarking.<n>Metriq integrates benchmark definition and execution, data collection, and public presentation into a unified workflow.<n>Using Metriq, we collect and publicly release results from more than ten quantum computers across multiple hardware vendors.
• Score: 33.7311511438168
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The fragmented landscape of quantum computer benchmarks, characterized by system-specific tools and inconsistent evaluation methodologies, hinders reliable cross-platform performance assessment. We introduce Metriq, an open-source collaborative platform for reproducible cross-platform quantum benchmarking that integrates benchmark definition and execution, data collection, and public presentation into a unified workflow. The Metriq benchmark suite spans both system-level metrics that characterize fundamental device properties such as entanglement quality, gate performance, and circuit speed, as well as application-inspired protocols that assess performance on quantum machine learning, optimization, and quantum simulation tasks. Benchmarks are chosen to scale with processor size, and the framework incorporates cost and resource estimation to support practical evaluation. Using Metriq, we collect and publicly release results from more than ten quantum computers across multiple hardware vendors, enabling systematic cross-platform comparison. The resulting curated dataset also reveals the practical strengths and limitations of individual benchmarks, creating a feedback loop that informs the ongoing refinement of the suite. To summarize performance across the benchmark suite, we introduce the Metriq Score, a composite index aggregating benchmark outcomes. We further present cross-benchmark analyses enabled by the shared dataset and their correlations with hardware calibration metrics. Through open development and data sharing, Metriq provides a practical foundation for reproducible benchmarking of quantum computers as hardware and benchmarking methods continue to evolve.

Related papers

• Flower Hub: A Reproducible Benchmarking Platform for Federated Learning in Simulation and Deployment [51.83961778298564]
  We present Flower Hub, a platform for publishing, discovering, and executing decentralized and federated applications.<n>We show how it enables reproducible benchmarking by packaging benchmarks as executable, versioned applications with standardized metadata, pinned dependencies, and explicit evaluation.
  arXiv  Detail & Related papers  (2026-08-25T20:09:44Z)
• Evaluation of Agents under Simulated AI Marketplace Dynamics [72.02953516461965]
  We introduce a simulation-based paradigm that evaluates information access systems as participants in a competitive marketplace.<n>We formalize the framework and outline a research agenda, motivated by business and economics, around marketplace simulation, metrics, optimization, and adoption in evaluation campaigns like TREC.
  arXiv  Detail & Related papers  (2026-04-15T15:01:36Z)
• Easy Data Unlearning Bench [53.1304932656586]
  We introduce a unified and benchmarking suite that simplifies the evaluation of unlearning algorithms.<n>By standardizing setup and metrics, it enables reproducible, scalable, and fair comparison across unlearning methods.
  arXiv  Detail & Related papers  (2026-02-18T12:20:32Z)
• QuSquare: Scalable Quality-Oriented Benchmark Suite for Pre-Fault-Tolerant Quantum Devices [0.19116784879310025]
  QuSquare consists of four benchmark tests that evaluate quantum hardware performance at both the system and application levels.<n>These benchmarks offer an integral, hardware-agnostic, and impartial methodology to quantify the quality and capabilities of current quantum computers.
  arXiv  Detail & Related papers  (2025-12-22T18:44:01Z)
• Featuremetric benchmarking: Quantum computer benchmarks based on circuit features [1.0842830860169255]
  Benchmarks that concisely summarize the performance of many-qubit quantum computers are essential for measuring progress towards the goal of useful quantum computation.<n>We present a benchmarking framework that is based on quantifying how a quantum computer's performance varies as a function of features of those circuits.
  arXiv  Detail & Related papers  (2025-04-17T01:49:02Z)
• Systematic benchmarking of quantum computers: status and recommendations [1.1961811541956795]
  Benchmarking is crucial for assessing the performance of quantum computers.<n>The document highlights key aspects such as component-level, system-level, software-level, HPC-level, and application-level benchmarks.
  arXiv  Detail & Related papers  (2025-03-06T19:05:13Z)
• Benchmarking quantum computers [0.0]
  Good benchmarks empower scientists, engineers, programmers, and users to understand a computing system's power. Bad benchmarks can misdirect research and inhibit progress. We discuss the role of benchmarks and benchmarking, and how good benchmarks can drive and measure progress.
  arXiv  Detail & Related papers  (2024-07-11T19:25:30Z)
• On-Chip Hardware-Aware Quantization for Mixed Precision Neural Networks [52.97107229149988]
  We propose an On-Chip Hardware-Aware Quantization framework, performing hardware-aware mixed-precision quantization on deployed edge devices. For efficiency metrics, we built an On-Chip Quantization Aware pipeline, which allows the quantization process to perceive the actual hardware efficiency of the quantization operator. For accuracy metrics, we propose Mask-Guided Quantization Estimation technology to effectively estimate the accuracy impact of operators in the on-chip scenario.
  arXiv  Detail & Related papers  (2023-09-05T04:39:34Z)
• QPack Scores: Quantitative performance metrics for application-oriented quantum computer benchmarking [1.0323063834827415]
  This paper presents the benchmark score definitions of QPack, an application-oriented cross-platform benchmarking suite for quantum computers and simulators. A comparison is made between various quantum computer simulators, running both locally and on vendors' remote cloud services.
  arXiv  Detail & Related papers  (2022-05-24T15:18:24Z)
• QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization [116.56171113972944]
  We show that carefully choosing the components of a QA-based metric is critical to performance. Our solution improves upon the best-performing entailment-based metric and achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2021-12-16T00:38:35Z)
• An Extensible Benchmark Suite for Learning to Simulate Physical Systems [60.249111272844374]
  We introduce a set of benchmark problems to take a step towards unified benchmarks and evaluation protocols. We propose four representative physical systems, as well as a collection of both widely used classical time-based and representative data-driven methods.
  arXiv  Detail & Related papers  (2021-08-09T17:39:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.