Quantum Computer Benchmarking: An Explorative Systematic Literature Review

• URL: http://arxiv.org/abs/2509.03078v1
• Date: Wed, 03 Sep 2025 07:13:52 GMT
• Title: Quantum Computer Benchmarking: An Explorative Systematic Literature Review
• Abstract summary: We conduct the most comprehensive systematic literature review of this area to date.<n>We develop a novel taxonomy and definitions for QC benchmarking, aligned with the quantum stack and its stakeholders.<n>Our analysis reveals recurring design patterns, exposes research gaps, and clarifies how benchmarking methods serve different stakeholders.
• Score: 2.2155582827470544
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: As quantum computing (QC) continues to evolve in hardware and software, measuring progress in this complex and diverse field remains a challenge. To track progress, uncover bottlenecks, and evaluate community efforts, benchmarks play a crucial role. But which benchmarking approach best addresses the diverse perspectives of QC stakeholders? We conducted the most comprehensive systematic literature review of this area to date, combining NLP-based clustering with expert analysis to develop a novel taxonomy and definitions for QC benchmarking, aligned with the quantum stack and its stakeholders. In addition to organizing benchmarks in distinct hardware, software, and application focused categories, our taxonomy hierarchically classifies benchmarking protocols in clearly defined subcategories. We develop standard terminology and map the interdependencies of benchmark categories to create a holistic, unified picture of the quantum benchmarking landscape. Our analysis reveals recurring design patterns, exposes research gaps, and clarifies how benchmarking methods serve different stakeholders. By structuring the field and providing a common language, our work offers a foundation for coherent benchmark development, fairer evaluation, and stronger cross-disciplinary collaboration in QC.

Related papers

• Measuring what matters: A scalable framework for application-level quantum benchmarking [0.21657999663857241]
  We present a scalable framework for application-level quantum benchmarking.<n>We present 13 benchmark families that reflect realistic workloads across multiple domains.<n>The benchmarks are designed to be reproducible, interpretable across stakeholder groups, and adaptable to evolving system capabilities.
  arXiv  Detail & Related papers  (2026-04-13T17:47:55Z)
• Beyond the Academic Monoculture: A Unified Framework and Industrial Perspective for Attributed Graph Clustering [75.50670592447219]
  Attributed Graph Clustering (AGC) is a fundamental unsupervised task that partitions nodes into cohesive groups by jointly modeling structural topology and node attributes.<n>This survey provides a comprehensive, industrially grounded review of AGC from three complementary perspectives.
  arXiv  Detail & Related papers  (2026-03-21T14:15:34Z)
• Metriq: A Collaborative Platform for Benchmarking Quantum Computers [33.7311511438168]
  We introduce Metriq, an open-source platform for reproducible cross-platform quantum benchmarking.<n>Metriq integrates benchmark definition and execution, data collection, and public presentation into a unified workflow.<n>Using Metriq, we collect and publicly release results from more than ten quantum computers across multiple hardware vendors.
  arXiv  Detail & Related papers  (2026-03-09T17:49:41Z)
• A Theoretical Framework for Adaptive Utility-Weighted Benchmarking [0.0]
  This paper introduces a theoretical framework that reconceptualizes benchmarking as a multilayer, adaptive network linking evaluation metrics, model components, and stakeholder groups through weighted interactions.<n>Using conjoint-derived utilities and a human-in-the-loop update rule, we formalize how human tradeoffs can be embedded into benchmark structure and how benchmarks can evolve dynamically while preserving stability and interpretability.
  arXiv  Detail & Related papers  (2026-02-12T19:33:47Z)
• A Comprehensive Survey on Benchmarks and Solutions in Software Engineering of LLM-Empowered Agentic System [56.40989626804489]
  This survey provides the first holistic analysis of Large Language Models-powered software engineering.<n>We review over 150 recent papers and propose a taxonomy along two key dimensions: (1) Solutions, categorized into prompt-based, fine-tuning-based, and agent-based paradigms, and (2) Benchmarks, including tasks such as code generation, translation, and repair.
  arXiv  Detail & Related papers  (2025-10-10T06:56:50Z)
• Context-Aware Hierarchical Taxonomy Generation for Scientific Papers via LLM-Guided Multi-Aspect Clustering [59.54662810933882]
  Existing taxonomy construction methods, leveraging unsupervised clustering or direct prompting of large language models, often lack coherence and granularity.<n>We propose a novel context-aware hierarchical taxonomy generation framework that integrates LLM-guided multi-aspect encoding with dynamic clustering.
  arXiv  Detail & Related papers  (2025-09-23T15:12:58Z)
• Expert Preference-based Evaluation of Automated Related Work Generation [54.29459509574242]
  We propose GREP, a multi-turn evaluation framework that integrates classical related work evaluation criteria with expert-specific preferences.<n>For better accessibility, we design two variants of GREP: a more precise variant with proprietary LLMs as evaluators, and a cheaper alternative with open-weight LLMs.
  arXiv  Detail & Related papers  (2025-08-11T13:08:07Z)
• Discrete Tokenization for Multimodal LLMs: A Comprehensive Survey [69.45421620616486]
  This work presents the first structured taxonomy and analysis of discrete tokenization methods designed for large language models (LLMs)<n>We categorize 8 representative VQ variants that span classical and modern paradigms and analyze their algorithmic principles, training dynamics, and integration challenges with LLM pipelines.<n>We identify key challenges including codebook collapse, unstable gradient estimation, and modality-specific encoding constraints.
  arXiv  Detail & Related papers  (2025-07-21T10:52:14Z)
• BenchQC -- Scalable and modular benchmarking of industrial quantum computing applications [26.629709879735532]
  BenchQC promotes an application-centric perspective for benchmarking real-world quantum applications.<n>We aim to uncover meaningful trends, provide systematic guidance on quantum utility, and distinguish promising research directions from less viable approaches.
  arXiv  Detail & Related papers  (2025-04-15T14:05:11Z)
• Systematic benchmarking of quantum computers: status and recommendations [1.1961811541956795]
  Benchmarking is crucial for assessing the performance of quantum computers.<n>The document highlights key aspects such as component-level, system-level, software-level, HPC-level, and application-level benchmarks.
  arXiv  Detail & Related papers  (2025-03-06T19:05:13Z)
• MultiAgentBench: Evaluating the Collaboration and Competition of LLM agents [59.825725526176655]
  Large Language Models (LLMs) have shown remarkable capabilities as autonomous agents.<n>Existing benchmarks either focus on single-agent tasks or are confined to narrow domains, failing to capture the dynamics of multi-agent coordination and competition.<n>We introduce MultiAgentBench, a benchmark designed to evaluate LLM-based multi-agent systems across diverse, interactive scenarios.
  arXiv  Detail & Related papers  (2025-03-03T05:18:50Z)
• Leveraging Large Language Models for NLG Evaluation: Advances and Challenges [57.88520765782177]
  Large Language Models (LLMs) have opened new avenues for assessing generated content quality, e.g., coherence, creativity, and context relevance. We propose a coherent taxonomy for organizing existing LLM-based evaluation metrics, offering a structured framework to understand and compare these methods. By discussing unresolved challenges, including bias, robustness, domain-specificity, and unified evaluation, this paper seeks to offer insights to researchers and advocate for fairer and more advanced NLG evaluation techniques.
  arXiv  Detail & Related papers  (2024-01-13T15:59:09Z)
• Investigating Fairness Disparities in Peer Review: A Language Model Enhanced Approach [77.61131357420201]
  We conduct a thorough and rigorous study on fairness disparities in peer review with the help of large language models (LMs) We collect, assemble, and maintain a comprehensive relational database for the International Conference on Learning Representations (ICLR) conference from 2017 to date. We postulate and study fairness disparities on multiple protective attributes of interest, including author gender, geography, author, and institutional prestige.
  arXiv  Detail & Related papers  (2022-11-07T16:19:42Z)
• QUARK: A Framework for Quantum Computing Application Benchmarking [0.0]
  We propose an application-centric benchmark method and the QUARK framework to foster the investigation and creation of application benchmarks for QC. This paper makes a case for application-level benchmarks and provides an in-depth "pen and paper" benchmark formulation of two reference problems.
  arXiv  Detail & Related papers  (2022-02-07T09:41:24Z)
• QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization [116.56171113972944]
  We show that carefully choosing the components of a QA-based metric is critical to performance. Our solution improves upon the best-performing entailment-based metric and achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2021-12-16T00:38:35Z)
• HAWKS: Evolving Challenging Benchmark Sets for Cluster Analysis [2.5329716878122404]
  Comprehensive benchmarking of clustering algorithms is difficult. There is no consensus regarding the best practice for rigorous benchmarking. We demonstrate the important role evolutionary algorithms play to support flexible generation of such benchmarks.
  arXiv  Detail & Related papers  (2021-02-13T15:01:34Z)
• Exploring and Analyzing Machine Commonsense Benchmarks [0.13999481573773073]
  We argue that the lack of a common vocabulary for aligning these approaches' metadata limits researchers in their efforts to understand systems' deficiencies. We describe our initial MCS Benchmark Ontology, an common vocabulary that formalizes benchmark metadata.
  arXiv  Detail & Related papers  (2020-12-21T19:01:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.