Evaluating LLM-based AI agents integrated with materials synthesis tools: the case of atomic layer deposition

• URL: http://arxiv.org/abs/2608.29309v1
• Date: Sat, 29 Aug 2026 14:51:17 GMT
• Title: Evaluating LLM-based AI agents integrated with materials synthesis tools: the case of atomic layer deposition
• Abstract summary: This work provides an overview of the different strategies that can be used to evaluate the performance of AI models and agents based on large language models (LLMs) for materials synthesis.
• Score: 0.013537117504260622
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This work provides an overview of the different strategies that can be used to evaluate the performance of AI models and agents based on large language models (LLMs) for materials synthesis. After providing a brief overview of the key technologies behind the current generation of AI agents based on LLMs, we summarize the different approaches to evaluating these models in the context of materials science and in particular on materials synthesis, with a specific emphasis on scenarios in which the models are directly integrated with experimental tools. We discuss evaluation strategies spanning knowledge and reasoning benchmarks, tool-use benchmarks, and closed loop benchmarks involving the interaction with experimental systems or realistic virtual tools. We use atomic layer deposition (ALD) as a case study, emphasizing how existing approaches in the literature both build from general approaches used beyond materials science and can be generalized to other materials synthesis techniques. Finally, we provide a practical evaluation framework to evaluate LLMs in the context of materials synthesis

Related papers

• Coupling Language Models with Physics-based Simulation for Synthesis of Inorganic Materials [31.895328442910937]
  We introduce a novel framework to evaluate Large Language Models (LLMs) in inorganic synthesis planning.<n>We focus on the niobium-oxygen system, which features multiple industrially relevant oxide phases with well-characterized data.<n>In our evaluation setting, classical search methods serve primarily as a foil rather than a direct competitor.
  arXiv  Detail & Related papers  (2026-05-29T19:41:29Z)
• PeroMAS: A Multi-agent System of Perovskite Material Discovery [51.859972927223936]
  Perovskite Solar Cells (PSCs) are renowned for their superior optoelectronic performance and cost potential.<n>Existing AI approaches focus predominantly on discrete models, including material design, process optimization, and property prediction.<n>We propose a multi-agent system for perovskite material discovery, named PeroMAS.
  arXiv  Detail & Related papers  (2026-02-10T09:33:06Z)
• Towards Agentic Intelligence for Materials Science [73.4576385477731]
  This survey advances a unique pipeline-centric view that spans from corpus curation and pretraining to goal-conditioned agents interfacing with simulation and experimental platforms.<n>To bridge communities and establish a shared frame of reference, we first present an integrated lens that aligns terminology, evaluation, and workflow stages across AI and materials science.
  arXiv  Detail & Related papers  (2026-01-29T23:48:43Z)
• LeMat-Synth: a multi-modal toolbox to curate broad synthesis procedure databases from scientific literature [60.879220305044726]
  We propose a multi-modal toolbox that employs large language models (LLMs) and vision language models (VLMs) to automatically extract and organize synthesis procedures and performance data.<n>We curated 81k open-access papers, yielding LeMat- Synth (v 1.0): a dataset containing synthesis procedures spanning 35 synthesis methods and 16 material classes.<n>We release a modular, open-source library designed to support community-driven extension to new corpora and synthesis domains.
  arXiv  Detail & Related papers  (2025-10-28T17:58:18Z)
• How do Large Language Models Understand Relevance? A Mechanistic Interpretability Perspective [64.00022624183781]
  Large language models (LLMs) can assess relevance and support information retrieval (IR) tasks.<n>We investigate how different LLM modules contribute to relevance judgment through the lens of mechanistic interpretability.
  arXiv  Detail & Related papers  (2025-04-10T16:14:55Z)
• Evaluating LLM-based Agents for Multi-Turn Conversations: A Survey [64.08485471150486]
  This survey examines evaluation methods for large language model (LLM)-based agents in multi-turn conversational settings.<n>We systematically reviewed nearly 250 scholarly sources, capturing the state of the art from various venues of publication.
  arXiv  Detail & Related papers  (2025-03-28T14:08:40Z)
• Towards Fully-Automated Materials Discovery via Large-Scale Synthesis Dataset and Expert-Level LLM-as-a-Judge [6.500470477634259]
  Our work aims to support the materials science community by providing a practical, data-driven resource.<n>We have curated a comprehensive dataset of 17K expert-verified synthesis recipes from open-access literature.<n>AlchemicalBench offers an end-to-end framework that supports research in large language models applied to synthesis prediction.
  arXiv  Detail & Related papers  (2025-02-23T06:16:23Z)
• Mining experimental data from Materials Science literature with Large Language Models: an evaluation study [1.9849264945671101]
  This study is dedicated to assessing the capabilities of large language models (LLMs) in extracting structured information from scientific documents in materials science. We focus on two critical tasks of information extraction: (i) a named entity recognition (NER) of studied materials and physical properties and (ii) a relation extraction (RE) between these entities. The performance of LLMs in executing these tasks is benchmarked against traditional models based on the BERT architecture and rule-based approaches (baseline)
  arXiv  Detail & Related papers  (2024-01-19T23:00:31Z)
• T-Eval: Evaluating the Tool Utilization Capability of Large Language Models Step by Step [69.64348626180623]
  Large language models (LLM) have achieved remarkable performance on various NLP tasks. How to evaluate and analyze the tool-utilization capability of LLMs is still under-explored. We introduce T-Eval to evaluate the tool utilization capability step by step.
  arXiv  Detail & Related papers  (2023-12-21T17:02:06Z)
• Artificial intelligence approaches for materials-by-design of energetic materials: state-of-the-art, challenges, and future directions [0.0]
  We review advances in AI-driven materials-by-design and their applications to energetic materials. We evaluate methods in the literature in terms of their capacity to learn from a small/limited number of data. We suggest a few promising future research directions for EM materials-by-design, such as meta-learning, active learning, Bayesian learning, and semi-/weakly-supervised learning.
  arXiv  Detail & Related papers  (2022-11-15T14:41:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.