Structured Chemistry Reasoning with Large Language Models

• URL: http://arxiv.org/abs/2311.09656v2
• Date: Fri, 9 Feb 2024 16:35:28 GMT
• Title: Structured Chemistry Reasoning with Large Language Models
• Authors: Siru Ouyang, Zhuosheng Zhang, Bing Yan, Xuan Liu, Yejin Choi, Jiawei Han, Lianhui Qin
• Abstract summary: Large Language Models (LLMs) excel in diverse areas, yet struggle with complex scientific reasoning, especially in chemistry. We introduce StructChem, a simple yet effective prompting strategy that offers the desired guidance and substantially boosts the LLMs' chemical reasoning capability. Tests across four chemistry areas -- quantum chemistry, mechanics, physical chemistry, and kinetics -- StructChem substantially enhances GPT-4's performance, with up to 30% peak improvement.
• Score: 70.13959639460015
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large Language Models (LLMs) excel in diverse areas, yet struggle with complex scientific reasoning, especially in the field of chemistry. Different from the simple chemistry tasks (e.g., molecule classification) addressed in previous studies, complex chemistry problems require not only vast knowledge and precise calculation, but also compositional reasoning about rich dynamic interactions of different concepts (e.g., temperature changes). Our study shows that even advanced LLMs, like GPT-4, can fail easily in different ways. Interestingly, the errors often stem not from a lack of domain knowledge within the LLMs, but rather from the absence of an effective reasoning structure that guides the LLMs to elicit the right knowledge, incorporate the knowledge in step-by-step reasoning, and iteratively refine results for further improved quality. On this basis, we introduce StructChem, a simple yet effective prompting strategy that offers the desired guidance and substantially boosts the LLMs' chemical reasoning capability. Testing across four chemistry areas -- quantum chemistry, mechanics, physical chemistry, and kinetics -- StructChem substantially enhances GPT-4's performance, with up to 30\% peak improvement. Our analysis also underscores the unique difficulties of precise grounded reasoning in science with LLMs, highlighting a need for more research in this area. Code is available at \url{https://github.com/ozyyshr/StructChem}.

Related papers

• MOOSE-Chem: Large Language Models for Rediscovering Unseen Chemistry Scientific Hypotheses [72.39144388083712]
  We propose an assumption that a majority of chemistry hypotheses can be resulted from a research background and several inspirations. To investigate these questions, we construct a benchmark consisting of 51 chemistry papers published in Nature, Science, or a similar level in 2024. Every paper is divided by chemistry PhD students into three components: background, inspirations, and hypothesis. The goal is to rediscover the hypothesis, given only the background and a large randomly selected chemistry literature corpus.
  arXiv  Detail & Related papers  (2024-10-09T17:19:58Z)
• ChemEval: A Comprehensive Multi-Level Chemical Evaluation for Large Language Models [62.37850540570268]
  Existing benchmarks in this domain fail to adequately meet the specific requirements of chemical research professionals. ChemEval identifies 4 crucial progressive levels in chemistry, assessing 12 dimensions of LLMs across 42 distinct chemical tasks. Results show that while general LLMs excel in literature understanding and instruction following, they fall short in tasks demanding advanced chemical knowledge.
  arXiv  Detail & Related papers  (2024-09-21T02:50:43Z)
• ChemVLM: Exploring the Power of Multimodal Large Language Models in Chemistry Area [50.15254966969718]
  We introduce textbfChemVLM, an open-source chemical multimodal large language model for chemical applications. ChemVLM is trained on a carefully curated bilingual dataset that enhances its ability to understand both textual and visual chemical information. We benchmark ChemVLM against a range of open-source and proprietary multimodal large language models on various tasks.
  arXiv  Detail & Related papers  (2024-08-14T01:16:40Z)
• ChemLLM: A Chemical Large Language Model [49.308528569982805]
  Large language models (LLMs) have made impressive progress in chemistry applications. However, the community lacks an LLM specifically designed for chemistry. Here, we introduce ChemLLM, a comprehensive framework that features the first LLM dedicated to chemistry.
  arXiv  Detail & Related papers  (2024-02-10T01:11:59Z)
• ChemDFM: A Large Language Foundation Model for Chemistry [27.864255196445324]
  A more generic and efficient solution would be an AI model that could address many tasks and support free-form dialogue in the broad field of chemistry. We develop ChemDFM, a pioneering LLM for chemistry trained on 34B tokens from chemical literature and textbooks, and fine-tuned using 2.7M instructions. We have open-sourced the inference codes, evaluation datasets, and model weights of ChemDFM on Huggingface.
  arXiv  Detail & Related papers  (2024-01-26T12:45:55Z)
• SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Models [70.5763210869525]
  We introduce an expansive benchmark suite SciBench for Large Language Model (LLM) SciBench contains a dataset featuring a range of collegiate-level scientific problems from mathematics, chemistry, and physics domains. The results reveal that the current LLMs fall short of delivering satisfactory performance, with the best overall score of merely 43.22%.
  arXiv  Detail & Related papers  (2023-07-20T07:01:57Z)
• What can Large Language Models do in chemistry? A comprehensive benchmark on eight tasks [41.9830989458936]
  Large Language Models (LLMs) with strong abilities in natural language processing tasks have emerged. We aim to evaluate capabilities of LLMs in a wide range of tasks across the chemistry domain.
  arXiv  Detail & Related papers  (2023-05-27T14:17:33Z)
• ChemCrow: Augmenting large-language models with chemistry tools [0.9195187117013247]
  Large-language models (LLMs) have shown strong performance in tasks across domains, but struggle with chemistry-related problems. In this study, we introduce ChemCrow, an LLM chemistry agent designed to accomplish tasks across organic synthesis, drug discovery, and materials design. Our agent autonomously planned and executed the syntheses of an insect repellent, three organocatalysts, and guided the discovery of a novel chromophore.
  arXiv  Detail & Related papers  (2023-04-11T17:41:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.