ORBIT-Q: Dual-axis benchmarking of autonomous agents in scientific quantum programming

• URL: http://arxiv.org/abs/2607.03105v1
• Date: Fri, 03 Jul 2026 08:39:46 GMT
• Title: ORBIT-Q: Dual-axis benchmarking of autonomous agents in scientific quantum programming
• Abstract summary: We introduce Open Research Benchmark for Integrated Tasks in Quantum Computing (ORBIT-Q) to address this gap.<n>At its core, ORBIT-Q contributes a carefully curated suite of complex, research-level quantum software frameworks.<n>We evaluate two efficiency dimensions: agent-side resource use and artifact-side runtime.
• Score: 0.8312877398178786
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Autonomous coding agents perform well on many conventional programming tasks, but scientific computing demands a rigorous validation paradigm that extends beyond simple functional test completion: generated code must preserve physical fidelity, differentiable workflows, framework-native semantics, and scalable representations. We introduce Open Research Benchmark for Integrated Tasks in Quantum Computing (ORBIT-Q) to address this gap. At its core, ORBIT-Q contributes a carefully curated suite of complex, research-level quantum workflows that serves as a challenging testbed for modern scientific programming. ORBIT-Q combines a rigorous multi-tier verification pipeline to support two orthogonal comparisons: different agent harness and model configurations at a fixed quantum software framework, and different quantum software frameworks at a fixed agent. In our systematic evaluations, TensorCircuit-NG (TC) exhibits the highest capability and performance efficiency among the evaluated quantum software frameworks under agent-driven programming, and Codex with GPT-5.5 is the strongest tested agent configuration on TC. However, a significant performance and design gap remains between frontier autonomous agents and human expert reference implementations. We further evaluate two efficiency dimensions: agent-side resource use and artifact-side runtime. Together, these results establish ORBIT-Q as a rigorous benchmark for autonomous scientific programming, framework-agent synergy, and quantum software performance.

Related papers

• Evaluating Verified Autonomy in Quantum Engineering [9.463290030138044]
  We introduce QIQCBench, a benchmark of $49$ expert-authored tasks spanning multiple layers including calibration and control, error correction and compilation, sensing and networking.<n>Across $17$ frontier agentic systems, QIQCBench reveals wide variation in verified performance.<n>These results expose a substantial gap between demonstrating capability and achieving reliable operation, and establish Quantum-Harbor as a foundation for measuring progress towards verified autonomy in quantum engineering.
  arXiv  Detail & Related papers  (2026-09-15T16:46:08Z)
• Lowering the implementation barrier of neutral-atom quantum computing with agentic workflows [40.12543056558646]
  Quantum computers are moving from research laboratories to industrial machines via the cloud and integrated into high-performance computing facilities.<n> translating theoretical quantum protocols into hardware experiments remains a major bottleneck, requiring expertise across protocol design, compilation, simulation, and cloud execution.<n>Here, we introduce an agentic workflow that automates this pipeline on neutral-atom quantum processors while keeping the researcher in the loop for critical validation.
  arXiv  Detail & Related papers  (2026-07-28T15:13:14Z)
• Quantum Software Engineering in Practice: FPGA and AI Integration for Quantum Certification [36.94429692322632]
  Quantum Software Engineering (QSE) responds to the need for systematic, disciplined, and quantifiable approaches to the development, operation, and maintenance of quantum software.<n>This paper presents QAccCert, a hybrid certification framework developed following QSE principles.<n>The framework implements entanglement certification through CHSH inequality violation in ideal quantum simulations using Qiskit AerSimulator.
  arXiv  Detail & Related papers  (2026-07-08T16:14:07Z)
• Benchmarking Quantum Software Testing with Scalable Quantum Programs [9.3695909584334]
  We present Qolumbina, a benchmark infrastructure for controlled QST experiments on scalable quantum programs.<n>Qlumbina curates 40 programs from open-source repositories, turns them into test-ready subjects.<n>We also propose QST-oriented criteria to characterize quantum programs along functionality, output behavior, development complexity, and quantum-specific execution complexity.
  arXiv  Detail & Related papers  (2026-07-02T10:56:35Z)
• A Model Context Protocol Server for Quantum Execution in Hybrid Quantum-HPC Environments [0.47078939738017017]
  Executing quantum algorithms requires generating code and managing complex computational resources.<n>We propose an AI-driven framework specifically designed to bridge this execution gap through the implementation of a Model Context Protocol server.
  arXiv  Detail & Related papers  (2026-04-09T14:50:56Z)
• MAS-Orchestra: Understanding and Improving Multi-Agent Reasoning Through Holistic Orchestration and Controlled Benchmarks [86.05918381895555]
  We propose MASOrchestra as a training-time framework that formulates MAS orchestration as a function-calling reinforcement learning problem.<n>In MAS-Orchestra, complex, goal-oriented subagents are abstracted as callable functions, enabling global reasoning over system structure.<n>Our analysis reveals that MAS gains depend critically on task structure, verification protocols, and the capabilities of both orchestrator and subagents.
  arXiv  Detail & Related papers  (2026-01-21T04:57:02Z)
• An Agentic Framework for Autonomous Materials Computation [70.24472585135929]
  Large Language Models (LLMs) have emerged as powerful tools for accelerating scientific discovery.<n>Recent advances integrate LLMs into agentic frameworks, enabling retrieval, reasoning, and tool use for complex scientific experiments.<n>Here, we present a domain-specialized agent designed for reliable automation of first-principles materials computations.
  arXiv  Detail & Related papers  (2025-12-22T15:03:57Z)
• QAgent: An LLM-based Multi-Agent System for Autonomous OpenQASM programming [8.73473101831257]
  We present QAgent, a multi-agent system that fully automates OpenQASM programming.<n>Our evaluations demonstrate substantial improvements across multiple LLMs of varying sizes.<n>We envision this multi-agent system as a key enabler for democratizing quantum programming, bridging expertise gaps, and accelerating the practical adoption of quantum computing.
  arXiv  Detail & Related papers  (2025-08-26T18:40:02Z)
• Context-Aware Unit Testing for Quantum Subroutines [14.117812847408523]
  Testing quantum software presents unique challenges due to the non-deterministic nature of quantum information, the high dimensionality of the underlying Hilbert space, complex hardware noise, and the inherent non-local properties of quantum systems.<n>We propose incorporating context-awareness into the testing process to address the computational complexity associated with unit testing in quantum systems.
  arXiv  Detail & Related papers  (2025-06-12T04:58:56Z)
• QCircuitBench: A Large-Scale Dataset for Benchmarking Quantum Algorithm Design [63.02824918725805]
  Quantum computing is recognized for the significant speedup it offers over classical computing through quantum algorithms.<n>QCircuitBench is the first benchmark dataset designed to evaluate AI's capability in designing and implementing quantum algorithms.
  arXiv  Detail & Related papers  (2024-10-10T14:24:30Z)
• A Design Framework for the Simulation of Distributed Quantum Computing [2.969582361376132]
  Growing demand for large-scale quantum computers is pushing research on Distributed Quantum Computing (DQC) Recent experimental efforts have demonstrated some of the building blocks for such a design. DQC systems are clusters of quantum processing units connected by means of quantum network infrastructures.
  arXiv  Detail & Related papers  (2023-06-20T13:52:05Z)
• QSAN: A Near-term Achievable Quantum Self-Attention Network [73.15524926159702]
  Self-Attention Mechanism (SAM) is good at capturing the internal connections of features. A novel Quantum Self-Attention Network (QSAN) is proposed for image classification tasks on near-term quantum devices.
  arXiv  Detail & Related papers  (2022-07-14T12:22:51Z)
• When BERT Meets Quantum Temporal Convolution Learning for Text Classification in Heterogeneous Computing [75.75419308975746]
  This work proposes a vertical federated learning architecture based on variational quantum circuits to demonstrate the competitive performance of a quantum-enhanced pre-trained BERT model for text classification. Our experiments on intent classification show that our proposed BERT-QTC model attains competitive experimental results in the Snips and ATIS spoken language datasets.
  arXiv  Detail & Related papers  (2022-02-17T09:55:21Z)
• QTRAN++: Improved Value Transformation for Cooperative Multi-Agent Reinforcement Learning [70.382101956278]
  QTRAN is a reinforcement learning algorithm capable of learning the largest class of joint-action value functions. Despite its strong theoretical guarantee, it has shown poor empirical performance in complex environments. We propose a substantially improved version, coined QTRAN++.
  arXiv  Detail & Related papers  (2020-06-22T05:08:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.