From Answers to States: Verifiable Process-Level Evaluation of Chemical Reasoning in Large Language Models

• URL: http://arxiv.org/abs/2606.03660v2
• Date: Wed, 03 Jun 2026 14:05:21 GMT
• Title: From Answers to States: Verifiable Process-Level Evaluation of Chemical Reasoning in Large Language Models
• Abstract summary: ChemCoTBench-V2 is a rule-verifiable diagnostic benchmark for evaluation of structured, verifier-addressable chemical reasoning traces.<n>It spans molecular understanding, molecule editing, molecular optimization, and reaction prediction, with 5,620 evaluation samples across 18 reporting tasks.
• Score: 37.34302729762671
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models are increasingly used as chemistry assistants, yet most chemistry benchmarks still score only final answers. This masks a critical failure mode: a model may output the correct molecule, product, or option while its reasoning violates chemical logic. Existing process-level evaluators are hard to scale because LLM judges and human step-level process annotation are costly, inconsistent, and vulnerable to hallucination. We introduce ChemCoTBench-V2, a rule-verifiable diagnostic benchmark for low-cost, auditable evaluation of structured, verifier-addressable chemical reasoning traces. It spans molecular understanding, molecule editing, molecular optimization, and reaction prediction, with 5,620 evaluation samples across 18 reporting tasks. Models must expose key intermediate steps in expert-designed templates, and those steps are checked with deterministic chemistry rules and, for closed-answer tasks, reference traces rather than another LLM judge. Open-ended molecular optimization is evaluated with oracle-verifiable state constraints rather than strict trace matching. The benchmark reports three separate signals: final-answer correctness, template adherence, and step-wise verifier correctness over expert-refined intermediate commitments. Experiments on frontier models reveal a persistent gap between final-answer success and structured-reasoning-state consistency: models often follow the requested format while failing chemical-step checks, or answer correctly with weak supporting reasoning. ChemCoTBench-V2 enables fine-grained model comparison and identifies the concrete step at which the trace first violates the verifier.

Related papers

• ChemDIRT: A Diversified Instruction, Representation, and Task Benchmark for Robust Chemistry-LLM Evaluation [18.06583682294473]
  ChemDIRT is a comprehensive evaluation framework designed to assess the robustness of chemical reasoning in large language models.<n>It measures performance across variations in instructions and molecular representations while spanning eight categories of chemistry tasks.<n>We benchmark a diverse set of open- and closed-source LLMs, revealing substantial prompt sensitivity, representation dependence, and uneven performance across task families.
  arXiv  Detail & Related papers  (2026-08-21T16:58:30Z)
• Controllable and Verifiable Process Data Synthesis for Process Reward Models [0.0]
  We propose a controllable and verifiable framework for synthesizing process supervision data for process reward models (PRMs)<n>Our framework first constructs a correct symbolic reasoning chain, injects a template-aware error into an intermediate step, recomputes subsequent steps under the corrupted state, and verifies that the injected step is not derivable from its prefix.<n>Experiments show that the synthesized data improve Best-of-8 reranking on logical reasoning benchmarks and transfer to mathematical reasoning.
  arXiv  Detail & Related papers  (2026-05-04T09:36:57Z)
• RxnNano:Training Compact LLMs for Chemical Reaction and Retrosynthesis Prediction via Hierarchical Curriculum Learning [20.904697361501174]
  We argue that the core challenge lies in instilling these knowledge into the models.<n>We propose a unified framework that prioritizes chemical understanding over scale through three key innovations.
  arXiv  Detail & Related papers  (2026-02-10T07:42:23Z)
• How well can off-the-shelf LLMs elucidate molecular structures from mass spectra using chain-of-thought reasoning? [51.286853421822705]
  Large language models (LLMs) have shown promise for reasoning-intensive scientific tasks, but their capability for chemical interpretation is still unclear.<n>We introduce a Chain-of-Thought (CoT) prompting framework and benchmark that evaluate how LLMs reason about mass spectral data to predict molecular structures.<n>Our evaluation across metrics of SMILES validity, formula consistency, and structural similarity reveals that while LLMs can produce syntactically valid and partially plausible structures, they fail to achieve chemical accuracy or link reasoning to correct molecular predictions.
  arXiv  Detail & Related papers  (2026-01-09T20:08:42Z)
• Atom-anchored LLMs speak Chemistry: A Retrosynthesis Demonstration [2.9496795797433073]
  We introduce a framework for molecular reasoning using general-purpose Large Language Models.<n>Our method anchors chain-of-thought reasoning to the molecular structure by using unique atomic identifiers.<n>Our work also provides a method to generate theoretically grounded synthetic datasets.
  arXiv  Detail & Related papers  (2025-10-18T17:27:44Z)
• Hard2Verify: A Step-Level Verification Benchmark for Open-Ended Frontier Math [80.46254366870447]
  We introduce Hard2Verify, a step-level verification benchmark produced with over 500 hours of human labor.<n>We evaluate 29 generative critics and process reward models, demonstrating that, beyond a few standouts, open-source verifiers lag closed source models.
  arXiv  Detail & Related papers  (2025-10-15T16:50:54Z)
• MolErr2Fix: Benchmarking LLM Trustworthiness in Chemistry via Modular Error Detection, Localization, Explanation, and Revision [22.708008313748824]
  Large Language Models (LLMs) have shown growing potential in molecular sciences, but they often produce chemically inaccurate descriptions.<n>This raises important concerns about their robustness and reliability in scientific applications.<n>We present the MolErr2Fix benchmark, designed to assess LLMs on error detection and correction in molecular descriptions.
  arXiv  Detail & Related papers  (2025-08-26T05:43:45Z)
• ChemActor: Enhancing Automated Extraction of Chemical Synthesis Actions with LLM-Generated Data [53.78763789036172]
  We present ChemActor, a fully fine-tuned large language model (LLM) as a chemical executor to convert between unstructured experimental procedures and structured action sequences.<n>This framework integrates a data selection module that selects data based on distribution divergence, with a general-purpose LLM, to generate machine-executable actions from a single molecule input.<n>Experiments on reaction-to-description (R2D) and description-to-action (D2A) tasks demonstrate that ChemActor achieves state-of-the-art performance, outperforming the baseline model by 10%.
  arXiv  Detail & Related papers  (2025-06-30T05:11:19Z)
• UAlign: Pushing the Limit of Template-free Retrosynthesis Prediction with Unsupervised SMILES Alignment [51.49238426241974]
  This paper introduces UAlign, a template-free graph-to-sequence pipeline for retrosynthesis prediction. By combining graph neural networks and Transformers, our method can more effectively leverage the inherent graph structure of molecules.
  arXiv  Detail & Related papers  (2024-03-25T03:23:03Z)
• Retrosynthesis Prediction with Conditional Graph Logic Network [118.70437805407728]
  Computer-aided retrosynthesis is finding renewed interest from both chemistry and computer science communities. We propose a new approach to this task using the Conditional Graph Logic Network, a conditional graphical model built upon graph neural networks.
  arXiv  Detail & Related papers  (2020-01-06T05:36:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.