Can Agents Price a Reaction? Evaluating LLMs on Chemical Cost Reasoning

• URL: http://arxiv.org/abs/2605.07251v1
• Date: Fri, 08 May 2026 05:19:59 GMT
• Title: Can Agents Price a Reaction? Evaluating LLMs on Chemical Cost Reasoning
• Abstract summary: Large Language Models (LLMs) have become increasingly capable as tool-using agents.<n>ChemCost is a benchmark of 1,427 evaluable reactions grounded to a frozen pricing snapshot covering 2,261 chemicals and 230,775 supplier quotes.<n>Experiments with frontier, open-weight, and chemistry-specialized LLM agents show that tool access is necessary but insufficient for solving the task.
• Score: 31.064444347894565
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large Language Models (LLMs) have become increasingly capable as tool-using agents, with benchmarks spanning diverse general agentic tasks. Yet rigorous evaluation of scientific tool use remains limited. In chemistry, recent agents can plan syntheses and invoke domain-specific tools, but evaluations often rely on curated demonstrations, expert assessment, or LLM-as-judge scoring rather than exact, judge-free ground truth. We address this gap with chemical procurement cost estimation, a practical task in which an agent must ground chemical identities, retrieve supplier quotes, select valid purchasable packs, normalize quantities, and compute cost from a reaction description. We introduce ChemCost, a benchmark of 1,427 evaluable reactions grounded to a frozen pricing snapshot covering 2,261 chemicals and 230,775 supplier quotes, supporting scalar scoring and stage-level diagnosis of grounding, retrieval, procurement, and arithmetic failures. To evaluate robustness, we further construct controlled noise-injected views that perturb chemical aliases, quantity expressions, missing fields, and input formatting. Experiments with frontier, open-weight, and chemistry-specialized LLM agents show that tool access is necessary but insufficient for solving the task. The strongest agents reach only 50.6% accuracy within 25% relative error on clean inputs and degrade substantially with realistic noise. Stage-level analysis further shows that failures arise from brittle parsing, ineffective evidence integration, invalid pack selection, and non-convergent tool use.

Related papers

• ChemDIRT: A Diversified Instruction, Representation, and Task Benchmark for Robust Chemistry-LLM Evaluation [18.06583682294473]
  ChemDIRT is a comprehensive evaluation framework designed to assess the robustness of chemical reasoning in large language models.<n>It measures performance across variations in instructions and molecular representations while spanning eight categories of chemistry tasks.<n>We benchmark a diverse set of open- and closed-source LLMs, revealing substantial prompt sensitivity, representation dependence, and uneven performance across task families.
  arXiv  Detail & Related papers  (2026-08-21T16:58:30Z)
• From Answers to States: Verifiable Process-Level Evaluation of Chemical Reasoning in Large Language Models [37.34302729762671]
  ChemCoTBench-V2 is a rule-verifiable diagnostic benchmark for evaluation of structured, verifier-addressable chemical reasoning traces.<n>It spans molecular understanding, molecule editing, molecular optimization, and reaction prediction, with 5,620 evaluation samples across 18 reporting tasks.
  arXiv  Detail & Related papers  (2026-06-02T13:47:19Z)
• SUPERChem: A Multimodal Reasoning Benchmark in Chemistry [47.60627566673109]
  SUPERChem is a benchmark of 500 expert-curated reasoning-intensive chemistry problems.<n>Each problem is paired with an expert-authored solution path.<n> Evaluations against a human baseline of 40.3% accuracy show that even the best-performing model, GPT-5 (High), reaches only 38.5%.
  arXiv  Detail & Related papers  (2025-12-01T04:46:35Z)
• Atom-anchored LLMs speak Chemistry: A Retrosynthesis Demonstration [2.9496795797433073]
  We introduce a framework for molecular reasoning using general-purpose Large Language Models.<n>Our method anchors chain-of-thought reasoning to the molecular structure by using unique atomic identifiers.<n>Our work also provides a method to generate theoretically grounded synthetic datasets.
  arXiv  Detail & Related papers  (2025-10-18T17:27:44Z)
• ChemOrch: Empowering LLMs with Chemical Intelligence via Synthetic Instructions [52.79349601462865]
  ChemOrch is a framework that synthesizes chemically grounded instruction-response pairs.<n>ChemOrch enables controllable diversity and levels of difficulty for the generated tasks.
  arXiv  Detail & Related papers  (2025-09-20T05:43:58Z)
• MolErr2Fix: Benchmarking LLM Trustworthiness in Chemistry via Modular Error Detection, Localization, Explanation, and Revision [22.708008313748824]
  Large Language Models (LLMs) have shown growing potential in molecular sciences, but they often produce chemically inaccurate descriptions.<n>This raises important concerns about their robustness and reliability in scientific applications.<n>We present the MolErr2Fix benchmark, designed to assess LLMs on error detection and correction in molecular descriptions.
  arXiv  Detail & Related papers  (2025-08-26T05:43:45Z)
• ChemActor: Enhancing Automated Extraction of Chemical Synthesis Actions with LLM-Generated Data [53.78763789036172]
  We present ChemActor, a fully fine-tuned large language model (LLM) as a chemical executor to convert between unstructured experimental procedures and structured action sequences.<n>This framework integrates a data selection module that selects data based on distribution divergence, with a general-purpose LLM, to generate machine-executable actions from a single molecule input.<n>Experiments on reaction-to-description (R2D) and description-to-action (D2A) tasks demonstrate that ChemActor achieves state-of-the-art performance, outperforming the baseline model by 10%.
  arXiv  Detail & Related papers  (2025-06-30T05:11:19Z)
• ChemAgent: Self-updating Library in Large Language Models Improves Chemical Reasoning [64.2106664137118]
  ChemAgent is a novel framework designed to improve the performance of large language models (LLMs)<n>It is developed by decomposing chemical tasks into sub-tasks and compiling these sub-tasks into a structured collection that can be referenced for future queries.<n>When presented with a new problem, ChemAgent retrieves and refines pertinent information from the library, which we call memory.
  arXiv  Detail & Related papers  (2025-01-11T17:10:30Z)
• Text-Augmented Multimodal LLMs for Chemical Reaction Condition Recommendation [38.76977853056086]
  Chemma-RC is a text-augmented multimodal LLM to identify effective conditions through task-specific dialogue and condition generation.<n>Chemma-RC learns a unified representation of chemical reactions by aligning multiple modalities-including text corpus, reaction SMILES, and reaction graphs-within a shared embedding module.<n>Performance benchmarking on datasets showed high precision in identifying optimal conditions, with up to 17% improvement over the current state-of-the-art methods.
  arXiv  Detail & Related papers  (2024-07-21T12:27:26Z)
• Unassisted Noise Reduction of Chemical Reaction Data Sets [59.127921057012564]
  We propose a machine learning-based, unassisted approach to remove chemically wrong entries from data sets. Our results show an improved prediction quality for models trained on the cleaned and balanced data sets.
  arXiv  Detail & Related papers  (2021-02-02T09:34:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.