QCoder Benchmark: Bridging Language Generation and Quantum Hardware through Simulator-Based Feedback

• URL: http://arxiv.org/abs/2510.26101v2
• Date: Sat, 01 Nov 2025 03:02:22 GMT
• Title: QCoder Benchmark: Bridging Language Generation and Quantum Hardware through Simulator-Based Feedback
• Authors: Taku Mikuriya, Tatsuya Ishigaki, Masayuki Kawarada, Shunya Minami, Tadashi Kadowaki, Yohichi Suzuki, Soshun Naito, Shunya Takata, Takumi Kato, Tamotsu Basseda, Reo Yamada, Hiroya Takamura,
• Abstract summary: We introduce QCoder Benchmark, an evaluation framework that assesses large language models (LLMs) on quantum programming.<n>Our benchmark supports evaluation using a quantum simulator environment beyond conventional Python execution.<n>Even advanced models like GPT-4o achieve only around 18.97% accuracy, highlighting the difficulty of the benchmark.<n>In contrast, reasoning-based models such as o3 reach up to 78% accuracy, outperforming averaged success rates of human-written codes.
• Score: 7.355017519768158
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models (LLMs) have increasingly been applied to automatic programming code generation. This task can be viewed as a language generation task that bridges natural language, human knowledge, and programming logic. However, it remains underexplored in domains that require interaction with hardware devices, such as quantum programming, where human coders write Python code that is executed on a quantum computer. To address this gap, we introduce QCoder Benchmark, an evaluation framework that assesses LLMs on quantum programming with feedback from simulated hardware devices. Our benchmark offers two key features. First, it supports evaluation using a quantum simulator environment beyond conventional Python execution, allowing feedback of domain-specific metrics such as circuit depth, execution time, and error classification, which can be used to guide better generation. Second, it incorporates human-written code submissions collected from real programming contests, enabling both quantitative comparisons and qualitative analyses of LLM outputs against human-written codes. Our experiments reveal that even advanced models like GPT-4o achieve only around 18.97% accuracy, highlighting the difficulty of the benchmark. In contrast, reasoning-based models such as o3 reach up to 78% accuracy, outperforming averaged success rates of human-written codes (39.98%). We release the QCoder Benchmark dataset and public evaluation API to support further research. (Codes and datasets are available at https://qcoder-bench.github.io/ )

Related papers

• EquiBench: Benchmarking Large Language Models' Reasoning about Program Semantics via Equivalence Checking [58.15568681219339]
  We introduce EquiBench, a new benchmark for evaluating large language models (LLMs)<n>This task directly tests a model's ability to reason about program semantics.<n>We evaluate 19 state-of-the-art LLMs and find that in the most challenging categories, the best accuracies are 63.8% and 76.2%, only modestly above the 50% random baseline.
  arXiv  Detail & Related papers  (2025-02-18T02:54:25Z)
• CRUXEval-X: A Benchmark for Multilingual Code Reasoning, Understanding and Execution [50.1875460416205]
  The CRUXEVAL-X code reasoning benchmark contains 19 programming languages.<n>It comprises at least 600 subjects for each language, along with 19K content-consistent tests in total.<n>Even a model trained solely on Python can achieve at most 34.4% Pass@1 in other languages.
  arXiv  Detail & Related papers  (2024-08-23T11:43:00Z)
• PythonSaga: Redefining the Benchmark to Evaluate Code Generating LLMs [1.9207412600219353]
  We evaluate two popular benchmarks for Python code generation, analyzing their diversity and difficulty. Our findings unveil a critical bias towards a limited set of programming concepts, neglecting most of the other concepts entirely. We propose a novel benchmark, PythonSaga, featuring 185 hand-crafted prompts on a balanced representation of 38 programming concepts.
  arXiv  Detail & Related papers  (2024-01-08T12:36:43Z)
• LeTI: Learning to Generate from Textual Interactions [60.425769582343506]
  We explore LMs' potential to learn from textual interactions (LETI) that not only check their correctness with binary labels but also pinpoint and explain errors in their outputs through textual feedback. Our focus is the code generation task, where the model produces code based on natural language instructions. LETI iteratively fine-tunes the model, using the objective LM, on a concatenation of natural language instructions, LM-generated programs, and textual feedback.
  arXiv  Detail & Related papers  (2023-05-17T15:53:31Z)
• Teaching Large Language Models to Self-Debug [62.424077000154945]
  Large language models (LLMs) have achieved impressive performance on code generation. We propose Self- Debugging, which teaches a large language model to debug its predicted program via few-shot demonstrations.
  arXiv  Detail & Related papers  (2023-04-11T10:43:43Z)
• Who Evaluates the Evaluators? On Automatic Metrics for Assessing AI-based Offensive Code Generators [1.7616042687330642]
  Code generators are an emerging solution for automatically writing programs starting from descriptions in natural language. In particular, code generators have been used for ethical hacking and offensive security testing by generating proof-of-concept attacks. This work analyzes a large set of output similarity metrics on offensive code generators.
  arXiv  Detail & Related papers  (2022-12-12T16:16:09Z)
• High-performance state-vector emulator of a gate-based quantum processor implemented in the Rust programming language [0.7366405857677227]
  We propose a high-performance state-vector emulator of a gate-based quantum processors developed in the Rust programming language. It supports OpenQASM 2.0 programming language for quantum circuits specification and has a user-friendly Python based API.
  arXiv  Detail & Related papers  (2022-09-23T07:56:24Z)
• Measuring Coding Challenge Competence With APPS [54.22600767666257]
  We introduce APPS, a benchmark for code generation. Our benchmark includes 10,000 problems, which range from having simple one-line solutions to being substantial algorithmic challenges. Recent models such as GPT-Neo can pass approximately 15% of the test cases of introductory problems.
  arXiv  Detail & Related papers  (2021-05-20T17:58:42Z)
• Extending Python for Quantum-Classical Computing via Quantum Just-in-Time Compilation [78.8942067357231]
  Python is a popular programming language known for its flexibility, usability, readability, and focus on developer productivity. We present a language extension to Python that enables heterogeneous quantum-classical computing via a robust C++ infrastructure for quantum just-in-time compilation.
  arXiv  Detail & Related papers  (2021-05-10T21:11:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.