oMeBench: Towards Robust Benchmarking of LLMs in Organic Mechanism Elucidation and Reasoning

• URL: http://arxiv.org/abs/2510.07731v2
• Date: Sun, 12 Oct 2025 08:15:32 GMT
• Title: oMeBench: Towards Robust Benchmarking of LLMs in Organic Mechanism Elucidation and Reasoning
• Authors: Ruiling Xu, Yifan Zhang, Qingyun Wang, Carl Edwards, Heng Ji,
• Abstract summary: We introduce oMeBench, the first large-scale, expert-curated benchmark for organic mechanism reasoning in organic chemistry.<n>We also propose oMeS, a dynamic evaluation framework that combines step-level logic and chemical similarity.
• Score: 44.36582860924775
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Organic reaction mechanisms are the stepwise elementary reactions by which reactants form intermediates and products, and are fundamental to understanding chemical reactivity and designing new molecules and reactions. Although large language models (LLMs) have shown promise in understanding chemical tasks such as synthesis design, it is unclear to what extent this reflects genuine chemical reasoning capabilities, i.e., the ability to generate valid intermediates, maintain chemical consistency, and follow logically coherent multi-step pathways. We address this by introducing oMeBench, the first large-scale, expert-curated benchmark for organic mechanism reasoning in organic chemistry. It comprises over 10,000 annotated mechanistic steps with intermediates, type labels, and difficulty ratings. Furthermore, to evaluate LLM capability more precisely and enable fine-grained scoring, we propose oMeS, a dynamic evaluation framework that combines step-level logic and chemical similarity. We analyze the performance of state-of-the-art LLMs, and our results show that although current models display promising chemical intuition, they struggle with correct and consistent multi-step reasoning. Notably, we find that using prompting strategy and fine-tuning a specialist model on our proposed dataset increases performance by 50% over the leading closed-source model. We hope that oMeBench will serve as a rigorous foundation for advancing AI systems toward genuine chemical reasoning.

Related papers

• ChemNavigator: Agentic AI Discovery of Design Rules for Organic Photocatalysts [0.0]
  ChemNavigator is an agentic AI system that autonomously derives structure-property relationships.<n>ChemNavigator autonomously identified six statistically significant design rules governing frontier orbital energies.
  arXiv  Detail & Related papers  (2026-01-23T07:44:28Z)
• How well can off-the-shelf LLMs elucidate molecular structures from mass spectra using chain-of-thought reasoning? [51.286853421822705]
  Large language models (LLMs) have shown promise for reasoning-intensive scientific tasks, but their capability for chemical interpretation is still unclear.<n>We introduce a Chain-of-Thought (CoT) prompting framework and benchmark that evaluate how LLMs reason about mass spectral data to predict molecular structures.<n>Our evaluation across metrics of SMILES validity, formula consistency, and structural similarity reveals that while LLMs can produce syntactically valid and partially plausible structures, they fail to achieve chemical accuracy or link reasoning to correct molecular predictions.
  arXiv  Detail & Related papers  (2026-01-09T20:08:42Z)
• A Scientific Reasoning Model for Organic Synthesis Procedure Generation [12.609346156252393]
  We present QFANG, a scientific reasoning language model capable of generating precise, structured experimental procedures.<n>We introduce a Chemistry-Guided Reasoning (CGR) framework that produces chain-of-thought data grounded in chemical knowledge at scale.<n>We apply Reinforcement Learning from Verifiable Rewards (RLVR) to further enhance procedural accuracy.
  arXiv  Detail & Related papers  (2025-12-15T18:55:39Z)
• ChemOrch: Empowering LLMs with Chemical Intelligence via Synthetic Instructions [52.79349601462865]
  ChemOrch is a framework that synthesizes chemically grounded instruction-response pairs.<n>ChemOrch enables controllable diversity and levels of difficulty for the generated tasks.
  arXiv  Detail & Related papers  (2025-09-20T05:43:58Z)
• ChemDFM-R: An Chemical Reasoner LLM Enhanced with Atomized Chemical Knowledge [14.6026550444088]
  This work focuses on the specific field of chemistry and develop a Chemical Reasoner LLM, ChemDFM-R.<n>We first construct a comprehensive dataset of atomized knowledge points to enhance the model's understanding of the fundamental principles and logical structure of chemistry.<n> Experiments on diverse chemical benchmarks demonstrate that ChemDFM-R achieves cutting-edge performance while providing interpretable, rationale-driven outputs.
  arXiv  Detail & Related papers  (2025-07-29T16:40:49Z)
• ChemActor: Enhancing Automated Extraction of Chemical Synthesis Actions with LLM-Generated Data [53.78763789036172]
  We present ChemActor, a fully fine-tuned large language model (LLM) as a chemical executor to convert between unstructured experimental procedures and structured action sequences.<n>This framework integrates a data selection module that selects data based on distribution divergence, with a general-purpose LLM, to generate machine-executable actions from a single molecule input.<n>Experiments on reaction-to-description (R2D) and description-to-action (D2A) tasks demonstrate that ChemActor achieves state-of-the-art performance, outperforming the baseline model by 10%.
  arXiv  Detail & Related papers  (2025-06-30T05:11:19Z)
• Beyond Chemical QA: Evaluating LLM's Chemical Reasoning with Modular Chemical Operations [43.623140005091535]
  We introduce ChemCoTBench, a reasoning framework that bridges molecular structure understanding with arithmetic-inspired operations.<n>ChemCoTBench formalizes chemical problem-solving into transparent, step-by-step reasoning.<n>We evaluate models on two high-impact tasks: Molecular Property Optimization and Chemical Reaction Prediction.
  arXiv  Detail & Related papers  (2025-05-27T15:15:44Z)
• Chemical reasoning in LLMs unlocks strategy-aware synthesis planning and reaction mechanism elucidation [0.3065062372337749]
  Large language models (LLMs) can serve as powerful tools enabling chemical analysis.<n>We leverage their ability to evaluate chemical strategies and guide search algorithms toward chemically meaningful solutions.<n>Our approach establishes a new paradigm for computer-aided chemistry.
  arXiv  Detail & Related papers  (2025-03-11T15:27:17Z)
• Learning Chemical Reaction Representation with Reactant-Product Alignment [50.28123475356234]
  RAlign is a novel chemical reaction representation learning model for various organic reaction-related tasks.<n>By integrating atomic correspondence between reactants and products, our model discerns the molecular transformations that occur during the reaction.<n>We introduce a reaction-center-aware attention mechanism that enables the model to concentrate on key functional groups.
  arXiv  Detail & Related papers  (2024-11-26T17:41:44Z)
• BatGPT-Chem: A Foundation Large Model For Retrosynthesis Prediction [65.93303145891628]
  BatGPT-Chem is a large language model with 15 billion parameters, tailored for enhanced retrosynthesis prediction. Our model captures a broad spectrum of chemical knowledge, enabling precise prediction of reaction conditions. This development empowers chemists to adeptly address novel compounds, potentially expediting the innovation cycle in drug manufacturing and materials science.
  arXiv  Detail & Related papers  (2024-08-19T05:17:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.