MatTools: Benchmarking Large Language Models for Materials Science Tools

• URL: http://arxiv.org/abs/2505.10852v1
• Date: Fri, 16 May 2025 04:43:05 GMT
• Title: MatTools: Benchmarking Large Language Models for Materials Science Tools
• Authors: Siyu Liu, Jiamin Xu, Beilin Ye, Bo Hu, David J. Srolovitz, Tongqi Wen,
• Abstract summary: MatTools is built on two complementary components: a materials simulation tool question-answer benchmark and a real-world tool-usage benchmark.<n>The QA benchmark comprises 69, QA225 pairs that assess the ability of an LLM to understand materials science tools.<n>The real-world benchmark contains 49 tasks (138 subtasks) requiring the generation of functional Python code for materials property calculations.
• Score: 5.876786336423598
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Large language models (LLMs) are increasingly applied to materials science questions, including literature comprehension, property prediction, materials discovery and alloy design. At the same time, a wide range of physics-based computational approaches have been developed in which materials properties can be calculated. Here, we propose a benchmark application to evaluate the proficiency of LLMs to answer materials science questions through the generation and safe execution of codes based on such physics-based computational materials science packages. MatTools is built on two complementary components: a materials simulation tool question-answer (QA) benchmark and a real-world tool-usage benchmark. We designed an automated methodology to efficiently collect real-world materials science tool-use examples. The QA benchmark, derived from the pymatgen (Python Materials Genomics) codebase and documentation, comprises 69,225 QA pairs that assess the ability of an LLM to understand materials science tools. The real-world benchmark contains 49 tasks (138 subtasks) requiring the generation of functional Python code for materials property calculations. Our evaluation of diverse LLMs yields three key insights: (1)Generalists outshine specialists;(2)AI knows AI; and (3)Simpler is better. MatTools provides a standardized framework for assessing and improving LLM capabilities for materials science tool applications, facilitating the development of more effective AI systems for materials science and general scientific research.

Related papers

• SPARQL Query Generation with LLMs: Measuring the Impact of Training Data Memorization and Knowledge Injection [81.78173888579941]
  Large Language Models (LLMs) are considered a well-suited method to increase the quality of the question-answering functionality.<n>LLMs are trained on web data, where researchers have no control over whether the benchmark or the knowledge graph was already included in the training data.<n>This paper introduces a novel method that evaluates the quality of LLMs by generating a SPARQL query from a natural-language question.
  arXiv  Detail & Related papers  (2025-07-18T12:28:08Z)
• Toward Greater Autonomy in Materials Discovery Agents: Unifying Planning, Physics, and Scientists [46.884317494606776]
  MAPPS consists of a Planner, a Tool Code Generator, and a Scientific Mediator.<n>By unifying planning, physics, and scientists, MAPPS enables flexible and reliable materials discovery with greater autonomy.
  arXiv  Detail & Related papers  (2025-06-05T22:07:06Z)
• Materials Generation in the Era of Artificial Intelligence: A Comprehensive Survey [54.40267149907223]
  Materials are the foundation of modern society, underpinning advancements in energy, electronics, healthcare, transportation, and infrastructure.<n>The ability to discover and design new materials with tailored properties is critical to solving some of the most pressing global challenges.<n>Data-driven generative models provide a powerful tool for materials design by directly create novel materials that satisfy predefined property requirements.
  arXiv  Detail & Related papers  (2025-05-22T08:33:21Z)
• DARWIN 1.5: Large Language Models as Materials Science Adapted Learners [46.7259033847682]
  We propose DARWIN 1.5, the largest open-source large language model tailored for materials science.<n> DARWIN eliminates the need for task-specific descriptors and enables a flexible, unified approach to material property prediction and discovery.<n>Our approach integrates 6M material domain papers and 21 experimental datasets from 49,256 materials across modalities while enabling cross-task knowledge transfer.
  arXiv  Detail & Related papers  (2024-12-16T16:51:27Z)
• HoneyComb: A Flexible LLM-Based Agent System for Materials Science [31.173615509567885]
  HoneyComb is the first large language model system specifically designed for materials science. MatSciKB is a curated, structured knowledge collection based on reliable literature. ToolHub employs an Inductive Tool Construction method to generate, decompose, and refine API tools for materials science.
  arXiv  Detail & Related papers  (2024-08-29T15:38:40Z)
• MMSci: A Dataset for Graduate-Level Multi-Discipline Multimodal Scientific Understanding [59.41495657570397]
  We present a comprehensive dataset compiled from Nature Communications articles covering 72 scientific fields.<n>We evaluated 19 proprietary and open-source models on two benchmark tasks, figure captioning and multiple-choice, and conducted human expert annotation.<n>Fine-tuning Qwen2-VL-7B with our task-specific data achieved better performance than GPT-4o and even human experts in multiple-choice evaluations.
  arXiv  Detail & Related papers  (2024-07-06T00:40:53Z)
• LLMatDesign: Autonomous Materials Discovery with Large Language Models [5.481299708562135]
  New materials can have significant scientific and technological implications. Recent advances in machine learning have enabled data-driven methods to rapidly screen or generate promising materials. We introduce LLMatDesign, a novel framework for interpretable materials design powered by large language models.
  arXiv  Detail & Related papers  (2024-06-19T02:35:02Z)
• What Are Tools Anyway? A Survey from the Language Model Perspective [67.18843218893416]
  Language models (LMs) are powerful yet mostly for text generation tasks. We provide a unified definition of tools as external programs used by LMs. We empirically study the efficiency of various tooling methods.
  arXiv  Detail & Related papers  (2024-03-18T17:20:07Z)
• Are LLMs Ready for Real-World Materials Discovery? [10.87312197950899]
  Large Language Models (LLMs) create exciting possibilities for powerful language processing tools to accelerate research in materials science. While LLMs have great potential to accelerate materials understanding and discovery, they currently fall short in being practical materials science tools. We show relevant failure cases of LLMs in materials science that reveal current limitations of LLMs related to comprehending and reasoning over complex, interconnected materials science knowledge.
  arXiv  Detail & Related papers  (2024-02-07T19:10:36Z)
• Agent-based Learning of Materials Datasets from Scientific Literature [0.0]
  We develop a chemist AI agent, powered by large language models (LLMs), to create structured datasets from natural language text. Our chemist AI agent, Eunomia, can plan and execute actions by leveraging the existing knowledge from decades of scientific research articles.
  arXiv  Detail & Related papers  (2023-12-18T20:29:58Z)
• CREATOR: Tool Creation for Disentangling Abstract and Concrete Reasoning of Large Language Models [74.22729793816451]
  Large Language Models (LLMs) have made significant progress in utilizing tools, but their ability is limited by API availability. We propose CREATOR, a novel framework that enables LLMs to create their own tools using documentation and code realization. We evaluate CREATOR on MATH and TabMWP benchmarks, respectively consisting of challenging math competition problems.
  arXiv  Detail & Related papers  (2023-05-23T17:51:52Z)
• Large Language Models as Master Key: Unlocking the Secrets of Materials Science with GPT [9.33544942080883]
  This article presents a new natural language processing (NLP) task called structured information inference (SII) to address the complexities of information extraction at the device level in materials science. We accomplished this task by tuning GPT-3 on an existing perovskite solar cell FAIR dataset with 91.8% F1-score and extended the dataset with data published since its release. We also designed experiments to predict the electrical performance of solar cells and design materials or devices with targeted parameters using large language models (LLMs)
  arXiv  Detail & Related papers  (2023-04-05T04:01:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.