Measuring what matters: A scalable framework for application-level quantum benchmarking

• URL: http://arxiv.org/abs/2604.11781v1
• Date: Mon, 13 Apr 2026 17:47:55 GMT
• Title: Measuring what matters: A scalable framework for application-level quantum benchmarking
• Abstract summary: We present a scalable framework for application-level quantum benchmarking.<n>We present 13 benchmark families that reflect realistic workloads across multiple domains.<n>The benchmarks are designed to be reproducible, interpretable across stakeholder groups, and adaptable to evolving system capabilities.
• Score: 0.21657999663857241
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: As quantum computing systems continue to mature, there is an increasing need for benchmarking methodologies that capture performance in terms of meaningful, application-level metrics. In this work, we present a scalable framework for application-level quantum benchmarking that is designed to support internal system evaluation and cross-platform comparison across technology providers. Our framework is guided by a set of core principles, including measurability, simplicity, scalability, and extensibility. We present 13 benchmark families that reflect realistic workloads across multiple domains. This enables the systematic evaluation of the quality of solutions, the total execution time, total used energy, as well as Time-to-Solution. The benchmarks are designed to be reproducible, interpretable across stakeholder groups, and adaptable to evolving system capabilities. The framework aims to bridge the gap between low-level performance metrics and real-world value, providing a unified approach to assessing quantum systems. The resulting benchmarks support development and validation and contribute to the foundation of industry-wide benchmarking standards.

Related papers

• Flower Hub: A Reproducible Benchmarking Platform for Federated Learning in Simulation and Deployment [51.83961778298564]
  We present Flower Hub, a platform for publishing, discovering, and executing decentralized and federated applications.<n>We show how it enables reproducible benchmarking by packaging benchmarks as executable, versioned applications with standardized metadata, pinned dependencies, and explicit evaluation.
  arXiv  Detail & Related papers  (2026-08-25T20:09:44Z)
• UniDial-EvalKit: A Unified Toolkit for Evaluating Multi-Faceted Conversational Abilities [70.79422099851506]
  We present UniDial-EvalKit (UDE), a unified evaluation toolkit for assessing interactive AI systems.<n>UDE standardizes heterogeneous data formats into a universal schema, streamlines complex evaluation pipelines through a modular architecture, and aligns metric calculations under a consistent scoring interface.
  arXiv  Detail & Related papers  (2026-03-24T13:01:31Z)
• Metriq: A Collaborative Platform for Benchmarking Quantum Computers [33.7311511438168]
  We introduce Metriq, an open-source platform for reproducible cross-platform quantum benchmarking.<n>Metriq integrates benchmark definition and execution, data collection, and public presentation into a unified workflow.<n>Using Metriq, we collect and publicly release results from more than ten quantum computers across multiple hardware vendors.
  arXiv  Detail & Related papers  (2026-03-09T17:49:41Z)
• A Theoretical Framework for Adaptive Utility-Weighted Benchmarking [0.0]
  This paper introduces a theoretical framework that reconceptualizes benchmarking as a multilayer, adaptive network linking evaluation metrics, model components, and stakeholder groups through weighted interactions.<n>Using conjoint-derived utilities and a human-in-the-loop update rule, we formalize how human tradeoffs can be embedded into benchmark structure and how benchmarks can evolve dynamically while preserving stability and interpretability.
  arXiv  Detail & Related papers  (2026-02-12T19:33:47Z)
• Benchmark^2: Systematic Evaluation of LLM Benchmarks [66.2731798872668]
  We propose Benchmark2, a comprehensive framework comprising three complementary metrics.<n>We conduct experiments across 15 benchmarks spanning mathematics, reasoning, and knowledge domains.<n>Our analysis reveals significant quality variations among existing benchmarks and demonstrates that selective benchmark construction can achieve comparable evaluation performance.
  arXiv  Detail & Related papers  (2026-01-07T14:59:03Z)
• Quantum Computer Benchmarking: An Explorative Systematic Literature Review [2.2155582827470544]
  We conduct the most comprehensive systematic literature review of this area to date.<n>We develop a novel taxonomy and definitions for QC benchmarking, aligned with the quantum stack and its stakeholders.<n>Our analysis reveals recurring design patterns, exposes research gaps, and clarifies how benchmarking methods serve different stakeholders.
  arXiv  Detail & Related papers  (2025-09-03T07:13:52Z)
• BenchQC -- Scalable and modular benchmarking of industrial quantum computing applications [26.629709879735532]
  BenchQC promotes an application-centric perspective for benchmarking real-world quantum applications.<n>We aim to uncover meaningful trends, provide systematic guidance on quantum utility, and distinguish promising research directions from less viable approaches.
  arXiv  Detail & Related papers  (2025-04-15T14:05:11Z)
• Systematic benchmarking of quantum computers: status and recommendations [1.1961811541956795]
  Benchmarking is crucial for assessing the performance of quantum computers.<n>The document highlights key aspects such as component-level, system-level, software-level, HPC-level, and application-level benchmarks.
  arXiv  Detail & Related papers  (2025-03-06T19:05:13Z)
• Benchmarks as Microscopes: A Call for Model Metrology [76.64402390208576]
  Modern language models (LMs) pose a new challenge in capability assessment. To be confident in our metrics, we need a new discipline of model metrology.
  arXiv  Detail & Related papers  (2024-07-22T17:52:12Z)
• ECBD: Evidence-Centered Benchmark Design for NLP [95.50252564938417]
  We propose Evidence-Centered Benchmark Design (ECBD), a framework which formalizes the benchmark design process into five modules. Each module requires benchmark designers to describe, justify, and support benchmark design choices. Our analysis reveals common trends in benchmark design and documentation that could threaten the validity of benchmarks' measurements.
  arXiv  Detail & Related papers  (2024-06-13T00:59:55Z)
• QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization [116.56171113972944]
  We show that carefully choosing the components of a QA-based metric is critical to performance. Our solution improves upon the best-performing entailment-based metric and achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2021-12-16T00:38:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.