Structured Chemistry Reasoning with Large Language Models
- URL: http://arxiv.org/abs/2311.09656v2
- Date: Fri, 9 Feb 2024 16:35:28 GMT
- Title: Structured Chemistry Reasoning with Large Language Models
- Authors: Siru Ouyang, Zhuosheng Zhang, Bing Yan, Xuan Liu, Yejin Choi, Jiawei
Han, Lianhui Qin
- Abstract summary: Large Language Models (LLMs) excel in diverse areas, yet struggle with complex scientific reasoning, especially in chemistry.
We introduce StructChem, a simple yet effective prompting strategy that offers the desired guidance and substantially boosts the LLMs' chemical reasoning capability.
Tests across four chemistry areas -- quantum chemistry, mechanics, physical chemistry, and kinetics -- StructChem substantially enhances GPT-4's performance, with up to 30% peak improvement.
- Score: 70.13959639460015
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Large Language Models (LLMs) excel in diverse areas, yet struggle with
complex scientific reasoning, especially in the field of chemistry. Different
from the simple chemistry tasks (e.g., molecule classification) addressed in
previous studies, complex chemistry problems require not only vast knowledge
and precise calculation, but also compositional reasoning about rich dynamic
interactions of different concepts (e.g., temperature changes). Our study shows
that even advanced LLMs, like GPT-4, can fail easily in different ways.
Interestingly, the errors often stem not from a lack of domain knowledge within
the LLMs, but rather from the absence of an effective reasoning structure that
guides the LLMs to elicit the right knowledge, incorporate the knowledge in
step-by-step reasoning, and iteratively refine results for further improved
quality. On this basis, we introduce StructChem, a simple yet effective
prompting strategy that offers the desired guidance and substantially boosts
the LLMs' chemical reasoning capability. Testing across four chemistry areas --
quantum chemistry, mechanics, physical chemistry, and kinetics -- StructChem
substantially enhances GPT-4's performance, with up to 30\% peak improvement.
Our analysis also underscores the unique difficulties of precise grounded
reasoning in science with LLMs, highlighting a need for more research in this
area. Code is available at \url{https://github.com/ozyyshr/StructChem}.
Related papers
- Are large language models superhuman chemists? [5.1611032009738205]
"ChemBench" is an automated framework designed to rigorously evaluate the chemical knowledge and reasoning abilities of state-of-the-art models.
We curated more than 7,000 question-answer pairs for a wide array of subfields of the chemical sciences.
We found that the best models outperformed the best human chemists in our study on average.
arXiv Detail & Related papers (2024-04-01T20:56:25Z) - GSM-Plus: A Comprehensive Benchmark for Evaluating the Robustness of LLMs as Mathematical Problem Solvers [68.77382332826167]
Large language models (LLMs) have achieved impressive performance across various mathematical reasoning benchmarks.
One essential and frequently occurring evidence is that when the math questions are slightly changed, LLMs can behave incorrectly.
This motivates us to evaluate the robustness of LLMs' math reasoning capability by testing a wide range of question variations.
arXiv Detail & Related papers (2024-02-29T15:26:14Z) - ChemLLM: A Chemical Large Language Model [49.308528569982805]
Large language models (LLMs) have made impressive progress in chemistry applications.
However, the community lacks an LLM specifically designed for chemistry.
Here, we introduce ChemLLM, a comprehensive framework that features the first LLM dedicated to chemistry.
arXiv Detail & Related papers (2024-02-10T01:11:59Z) - ChemDFM: Dialogue Foundation Model for Chemistry [27.804229420333137]
ChemDFM-13B is trained on 34B tokens from chemical literature, textbooks, and instructions as well as various data from the general domain.
It can store, understand, and reason over chemical knowledge and languages while still possessing advanced free-form language comprehension capabilities.
ChemDFM can also surpass GPT-4 on a great portion of chemical tasks, despite the significant size difference.
arXiv Detail & Related papers (2024-01-26T12:45:55Z) - SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Models [70.5763210869525]
We introduce an expansive benchmark suite SciBench for Large Language Model (LLM)
SciBench contains a dataset featuring a range of collegiate-level scientific problems from mathematics, chemistry, and physics domains.
The results reveal that the current LLMs fall short of delivering satisfactory performance, with the best overall score of merely 43.22%.
arXiv Detail & Related papers (2023-07-20T07:01:57Z) - Empowering Molecule Discovery for Molecule-Caption Translation with Large Language Models: A ChatGPT Perspective [53.300288393173204]
Large Language Models (LLMs) have shown remarkable performance in various cross-modal tasks.
In this work, we propose an In-context Few-Shot Molecule Learning paradigm for molecule-caption translation.
We evaluate the effectiveness of MolReGPT on molecule-caption translation, including molecule understanding and text-based molecule generation.
arXiv Detail & Related papers (2023-06-11T08:16:25Z) - What can Large Language Models do in chemistry? A comprehensive
benchmark on eight tasks [41.9830989458936]
Large Language Models (LLMs) with strong abilities in natural language processing tasks have emerged.
We aim to evaluate capabilities of LLMs in a wide range of tasks across the chemistry domain.
arXiv Detail & Related papers (2023-05-27T14:17:33Z) - ChemCrow: Augmenting large-language models with chemistry tools [0.9195187117013247]
Large-language models (LLMs) have shown strong performance in tasks across domains, but struggle with chemistry-related problems.
In this study, we introduce ChemCrow, an LLM chemistry agent designed to accomplish tasks across organic synthesis, drug discovery, and materials design.
Our agent autonomously planned and executed the syntheses of an insect repellent, three organocatalysts, and guided the discovery of a novel chromophore.
arXiv Detail & Related papers (2023-04-11T17:41:13Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.