OmniMatBench: A Human-Calibrated Multimodal Reasoning Benchmark Across 19 Materials Science Subfields

• URL: http://arxiv.org/abs/2605.29833v2
• Date: Fri, 29 May 2026 03:29:38 GMT
• Title: OmniMatBench: A Human-Calibrated Multimodal Reasoning Benchmark Across 19 Materials Science Subfields
• Abstract summary: We present OmniMatBench, a human-calibrated multimodal reasoning benchmark for materials science.<n>We evaluate 13 open-source and closed-source MLLMs and find that the best model achieves a 0.372 overall score.
• Score: 61.92908843275242
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: As multimodal language models play an increasingly important role in scientific research, materials science offers a critical testbed due to its interdisciplinary, multimodal, and application-driven nature. However, existing materials benchmarks mainly focus on property prediction, knowledge QA, or characterization understanding, leaving the broader reasoning process from materials knowledge to application underexplored. To fill this gap, we present OmniMatBench, a human-calibrated multimodal reasoning benchmark for materials science. OmniMatBench contains 3,171 expert-curated QA and calculation problems across 19 materials-science subfields, spanning fundamental materials knowledge, structural and engineering materials, materials processing and manufacturing, and functional and applied materials. We evaluate 13 open-source and closed-source MLLMs and find that the best model achieves only a 0.372 overall score, revealing a substantial gap in current materials-science reasoning. Further analysis shows strong variation across subfields, fixed reasoning heuristics, uneven materials knowledge, and limited high-level knowledge application under formula-, retrieval-, and code-assisted settings. OmniMatBench provides crucial insights into the capabilities and limitations of current MLLMs and establishes a foundation for reliable AI assistants in materials-science research.

Related papers

• MatSciBench: Benchmarking the Reasoning Ability of Large Language Models in Materials Science [28.11660982198711]
  MatSciBench is a comprehensive college-level benchmark comprising 1,340 problems that span the essential subdisciplines of materials science.<n>MatSciBench features a structured and fine-grained taxonomy that categorizes materials science questions into 6 primary fields and 31 sub-fields.<n> Evaluations of leading models reveal that even the highest-performing model, Gemini-2.5-Pro, achieves under 80% accuracy on college-level materials science questions.
  arXiv  Detail & Related papers  (2025-10-14T05:59:40Z)
• Can Multimodal LLMs See Materials Clearly? A Multimodal Benchmark on Materials Characterization [31.165896296600334]
  MatCha is the first benchmark for materials characterization image understanding.<n>MatCha comprises 1,500 questions that demand expert-level domain expertise.<n>Our evaluation of state-of-the-art MLLMs on MatCha reveals a significant performance gap compared to human experts.
  arXiv  Detail & Related papers  (2025-09-11T09:50:16Z)
• Materials Generation in the Era of Artificial Intelligence: A Comprehensive Survey [54.40267149907223]
  Materials are the foundation of modern society, underpinning advancements in energy, electronics, healthcare, transportation, and infrastructure.<n>The ability to discover and design new materials with tailored properties is critical to solving some of the most pressing global challenges.<n>Data-driven generative models provide a powerful tool for materials design by directly create novel materials that satisfy predefined property requirements.
  arXiv  Detail & Related papers  (2025-05-22T08:33:21Z)
• MatTools: Benchmarking Large Language Models for Materials Science Tools [5.876786336423598]
  MatTools is built on two complementary components: a materials simulation tool question-answer benchmark and a real-world tool-usage benchmark.<n>The QA benchmark comprises 69, QA225 pairs that assess the ability of an LLM to understand materials science tools.<n>The real-world benchmark contains 49 tasks (138 subtasks) requiring the generation of functional Python code for materials property calculations.
  arXiv  Detail & Related papers  (2025-05-16T04:43:05Z)
• DARWIN 1.5: Large Language Models as Materials Science Adapted Learners [46.7259033847682]
  We propose DARWIN 1.5, the largest open-source large language model tailored for materials science.<n> DARWIN eliminates the need for task-specific descriptors and enables a flexible, unified approach to material property prediction and discovery.<n>Our approach integrates 6M material domain papers and 21 experimental datasets from 49,256 materials across modalities while enabling cross-task knowledge transfer.
  arXiv  Detail & Related papers  (2024-12-16T16:51:27Z)
• Foundation Model for Composite Microstructures: Reconstruction, Stiffness, and Nonlinear Behavior Prediction [0.0]
  We present the Material Masked Autoencoder (MMAE), a self-supervised Vision Transformer pretrained on a large corpus of short-fiber composite images.<n>We demonstrate two key applications: (i) predicting homogenized stiffness components through fine-tuning on limited data, and (ii) inferring physically interpretable parameters by coupling MMAE with an interaction-based material network.
  arXiv  Detail & Related papers  (2024-11-10T19:06:25Z)
• Are LLMs Ready for Real-World Materials Discovery? [10.87312197950899]
  Large Language Models (LLMs) create exciting possibilities for powerful language processing tools to accelerate research in materials science. While LLMs have great potential to accelerate materials understanding and discovery, they currently fall short in being practical materials science tools. We show relevant failure cases of LLMs in materials science that reveal current limitations of LLMs related to comprehending and reasoning over complex, interconnected materials science knowledge.
  arXiv  Detail & Related papers  (2024-02-07T19:10:36Z)
• Multimodal Foundation Models for Material Property Prediction and Discovery [7.167520424757711]
  We introduce Multimodal Learning for Materials (MultiMat), which enables self-supervised multi-modality training of foundation models for materials.<n>We demonstrate MultiMat's potential using data from the Materials Project database on multiple axes.
  arXiv  Detail & Related papers  (2023-11-30T18:35:29Z)
• SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Models [70.5763210869525]
  We introduce an expansive benchmark suite SciBench for Large Language Model (LLM) SciBench contains a dataset featuring a range of collegiate-level scientific problems from mathematics, chemistry, and physics domains. The results reveal that the current LLMs fall short of delivering satisfactory performance, with the best overall score of merely 43.22%.
  arXiv  Detail & Related papers  (2023-07-20T07:01:57Z)
• Artificial Intelligence in Concrete Materials: A Scientometric View [77.34726150561087]
  This chapter aims to uncover the main research interests and knowledge structure of the existing literature on AI for concrete materials. To begin with, a total of 389 journal articles published from 1990 to 2020 were retrieved from the Web of Science. Scientometric tools such as keyword co-occurrence analysis and documentation co-citation analysis were adopted to quantify features and characteristics of the research field.
  arXiv  Detail & Related papers  (2022-09-17T18:24:56Z)
• Machine Learning in Nano-Scale Biomedical Engineering [77.75587007080894]
  We review the existing research regarding the use of machine learning in nano-scale biomedical engineering. The main challenges that can be formulated as ML problems are classified into the three main categories. For each of the presented methodologies, special emphasis is given to its principles, applications, and limitations.
  arXiv  Detail & Related papers  (2020-08-05T15:45:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.