Can LLMs Solve Science or Just Write Code? Evaluating Quantum Solver Generation

• URL: http://arxiv.org/abs/2605.07525v2
• Date: Mon, 11 May 2026 15:11:44 GMT
• Title: Can LLMs Solve Science or Just Write Code? Evaluating Quantum Solver Generation
• Abstract summary: Large Language Models (LLMs) show strong capabilities in code generation, motivating their use in automated quantum solver development.<n>This work introduces Q-SAGE, an iterative methodology to evaluate LLMs' capability in generating solvers for scientific problems.
• Score: 4.121927985291393
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large Language Models (LLMs) show strong capabilities in code generation, motivating their use in automated quantum solver development. However, in quantum computing, successful execution of generated code is not sufficient: correctness depends on numerically accurate results, which are sensitive to non-trivial mappings, hybrid quantum-classical workflows, and algorithm-specific approximations. This work introduces Q-SAGE, an iterative methodology to evaluate LLMs' capability in generating quantum solvers for scientific problems. The methodology adopts an iterative approach by executing the script generated by the LLM, comparing the result with the result of a classical solver, and refining the script until the two results match within a tolerance threshold. We empirically evaluated the methodology with five families of scientific problems of different complexities and five LLMs, both open source and proprietary. The results show that iterative refinement substantially improves success rates, but introduces a significant computational overhead. Moreover, as model capability increases, failure modes shift from execution errors to numerical inaccuracies, highlighting the current limitations of LLM-based quantum software.

Related papers

• QBugLM: An Agentic Benchmarking Framework for LLM-based Quantum Software Debugging [0.8609132348927196]
  Large language models (LLMs) have shown strong performance on classical software engineering tasks.<n>We propose QBugLM, a multi-agent framework that automates the quantum software debug pipeline.
  arXiv  Detail & Related papers  (2026-06-05T14:34:46Z)
• Beyond Rules: LLM-Powered Linting for Quantum Programs [4.968531828144473]
  We introduce LintQ-LLM+CoT and LintQ-LLM+RAG, novel approaches that redefine the detection of quantum programming problems.<n>LLMs specialized via Chain-of-Thought (CoT) prompting and a Retrieval-Augmented Generation (RAG) system grounds the model's reasoning in a curated knowledge base.<n>Our results show that LLM-based approaches, with and without RAG, outperform LintQ in terms of quantum programming problems detection correctness (precision) and completeness (recall)
  arXiv  Detail & Related papers  (2026-05-05T16:31:14Z)
• QuanBench: Benchmarking Quantum Code Generation with Large Language Models [7.807551490308163]
  Large language models (LLMs) have demonstrated good performance in general code generation.<n>This paper presents QuanBench, a benchmark for evaluating LLMs on quantum code generation.
  arXiv  Detail & Related papers  (2025-10-19T10:08:36Z)
• SciML Agents: Write the Solver, Not the Solution [69.5021018644143]
  We introduce two new datasets: a diagnostic dataset of adversarial "misleading" problems; and a large-scale benchmark of 1,000 diverse ODE tasks.<n>We evaluate open- and closed-source LLM models along two axes: (i) unguided versus guided prompting with domain-specific knowledge; and (ii) off-the-shelf versus fine-tuned variants.<n>Preliminary results indicate that careful prompting and fine-tuning can yield a specialized LLM agent capable of reliably solving simple ODE problems.
  arXiv  Detail & Related papers  (2025-09-12T02:53:57Z)
• Quantum Knowledge Distillation for Large Language Models [10.023534560183919]
  We propose a Quantum knowledge Distillation model for Large Language Models (QD-LLM)<n>In classical simulation, QD-LLM outperforms several mainstream distillation methods on multiple text classification tasks.<n>We deploy the obtained circuits on the Baihua superconducting quantum processor via the Quafu platform to assess practical feasibility.
  arXiv  Detail & Related papers  (2025-05-19T14:56:24Z)
• Quantizing Large Language Models for Code Generation: A Differentiated Replication [51.85505914274633]
  Large Language Models (LLMs) have shown an impressive capability in code generation and, specifically, to automatically implement requirements described in natural language.<n>LLMs pose significant challenges related to their memory (and, consequently, carbon) footprint.<n>New frontier for LLM quantization is 4-bit precision, resulting in an average memory footprint reduction of 70%.
  arXiv  Detail & Related papers  (2025-03-10T09:26:08Z)
• LLM2: Let Large Language Models Harness System 2 Reasoning [65.89293674479907]
  Large language models (LLMs) have exhibited impressive capabilities across a myriad of tasks, yet they occasionally yield undesirable outputs.<n>We introduce LLM2, a novel framework that combines an LLM with a process-based verifier.<n>LLMs2 is responsible for generating plausible candidates, while the verifier provides timely process-based feedback to distinguish desirable and undesirable outputs.
  arXiv  Detail & Related papers  (2024-12-29T06:32:36Z)
• Precision or Peril: Evaluating Code Quality from Quantized Large Language Models [0.5249805590164902]
  Quantization has emerged as a way to mitigate the memory overhead of Large Language Models. This study aims to evaluate the current code generation capabilities of smaller LLMs using various metrics.
  arXiv  Detail & Related papers  (2024-11-16T01:31:29Z)
• Improving LLM Reasoning through Scaling Inference Computation with Collaborative Verification [52.095460362197336]
  Large language models (LLMs) struggle with consistent and accurate reasoning. LLMs are trained primarily on correct solutions, reducing their ability to detect and learn from errors. We propose a novel collaborative method integrating Chain-of-Thought (CoT) and Program-of-Thought (PoT) solutions for verification.
  arXiv  Detail & Related papers  (2024-10-05T05:21:48Z)
• What's Wrong with Your Code Generated by Large Language Models? An Extensive Study [92.62952504133926]
  This study evaluated the performance of three leading closed-source LLMs and six popular open-source LLMs on three commonly used benchmarks.<n>We developed a taxonomy of bugs for incorrect codes and analyzed the root cause for common bug types.<n>We propose a novel training-free iterative method that introduces self-critique, enabling LLMs to critique and correct their generated code.
  arXiv  Detail & Related papers  (2024-07-08T17:27:17Z)
• LLMC: Benchmarking Large Language Model Quantization with a Versatile Compression Toolkit [55.73370804397226]
  Quantization, a key compression technique, can effectively mitigate these demands by compressing and accelerating large language models. We present LLMC, a plug-and-play compression toolkit, to fairly and systematically explore the impact of quantization. Powered by this versatile toolkit, our benchmark covers three key aspects: calibration data, algorithms (three strategies), and data formats.
  arXiv  Detail & Related papers  (2024-05-09T11:49:05Z)
• LEVER: Learning to Verify Language-to-Code Generation with Execution [64.36459105535]
  We propose LEVER, a simple approach to improve language-to-code generation by learning to verify the generated programs with their execution results. Specifically, we train verifiers to determine whether a program sampled from the LLMs is correct or not based on the natural language input, the program itself and its execution results. LEVER consistently improves over the base code LLMs(4.6% to 10.9% with code-davinci) and achieves new state-of-the-art results on all of them.
  arXiv  Detail & Related papers  (2023-02-16T18:23:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.