Can Multimodal LLMs See Materials Clearly? A Multimodal Benchmark on Materials Characterization

• URL: http://arxiv.org/abs/2509.09307v1
• Date: Thu, 11 Sep 2025 09:50:16 GMT
• Title: Can Multimodal LLMs See Materials Clearly? A Multimodal Benchmark on Materials Characterization
• Authors: Zhengzhao Lai, Youbin Zheng, Zhenyang Cai, Haonan Lyu, Jinpu Yang, Hongqing Liang, Yan Hu, Benyou Wang,
• Abstract summary: MatCha is the first benchmark for materials characterization image understanding.<n>MatCha comprises 1,500 questions that demand expert-level domain expertise.<n>Our evaluation of state-of-the-art MLLMs on MatCha reveals a significant performance gap compared to human experts.
• Score: 31.165896296600334
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Materials characterization is fundamental to acquiring materials information, revealing the processing-microstructure-property relationships that guide material design and optimization. While multimodal large language models (MLLMs) have recently shown promise in generative and predictive tasks within materials science, their capacity to understand real-world characterization imaging data remains underexplored. To bridge this gap, we present MatCha, the first benchmark for materials characterization image understanding, comprising 1,500 questions that demand expert-level domain expertise. MatCha encompasses four key stages of materials research comprising 21 distinct tasks, each designed to reflect authentic challenges faced by materials scientists. Our evaluation of state-of-the-art MLLMs on MatCha reveals a significant performance gap compared to human experts. These models exhibit degradation when addressing questions requiring higher-level expertise and sophisticated visual perception. Simple few-shot and chain-of-thought prompting struggle to alleviate these limitations. These findings highlight that existing MLLMs still exhibit limited adaptability to real-world materials characterization scenarios. We hope MatCha will facilitate future research in areas such as new material discovery and autonomous scientific agents. MatCha is available at https://github.com/FreedomIntelligence/MatCha.

Related papers

• Materials Generation in the Era of Artificial Intelligence: A Comprehensive Survey [54.40267149907223]
  Materials are the foundation of modern society, underpinning advancements in energy, electronics, healthcare, transportation, and infrastructure.<n>The ability to discover and design new materials with tailored properties is critical to solving some of the most pressing global challenges.<n>Data-driven generative models provide a powerful tool for materials design by directly create novel materials that satisfy predefined property requirements.
  arXiv  Detail & Related papers  (2025-05-22T08:33:21Z)
• How do Large Language Models Understand Relevance? A Mechanistic Interpretability Perspective [64.00022624183781]
  Large language models (LLMs) can assess relevance and support information retrieval (IR) tasks.<n>We investigate how different LLM modules contribute to relevance judgment through the lens of mechanistic interpretability.
  arXiv  Detail & Related papers  (2025-04-10T16:14:55Z)
• Towards Visual Text Grounding of Multimodal Large Language Model [88.0588924255417]
  We introduce TRIG, a novel task with a newly designed instruction dataset for benchmarking text-rich image grounding.<n>Specifically, we propose an OCR-LLM-human interaction pipeline to create 800 manually annotated question-answer pairs as a benchmark.<n>A comprehensive evaluation of various MLLMs on our proposed benchmark exposes substantial limitations in their grounding capability on text-rich images.
  arXiv  Detail & Related papers  (2025-04-07T12:01:59Z)
• EmbodiedEval: Evaluate Multimodal LLMs as Embodied Agents [57.4686961979566]
  EmbodiedEval is a comprehensive and interactive evaluation benchmark for MLLMs with embodied tasks.<n>It covers a broad spectrum of existing embodied AI tasks with significantly enhanced diversity.<n>We evaluated the state-of-the-art MLLMs on EmbodiedEval and found that they have a significant shortfall compared to human level on embodied tasks.
  arXiv  Detail & Related papers  (2025-01-21T03:22:10Z)
• MatExpert: Decomposing Materials Discovery by Mimicking Human Experts [26.364419690908992]
  MatExpert is a novel framework that leverages Large Language Models and contrastive learning to accelerate the discovery and design of new solid-state materials. Inspired by the workflow of human materials design experts, our approach integrates three key stages: retrieval, transition, and generation. MatExpert represents a meaningful advancement in computational material discovery using langauge-based generative models.
  arXiv  Detail & Related papers  (2024-10-26T00:44:54Z)
• MC-Bench: A Benchmark for Multi-Context Visual Grounding in the Era of MLLMs [61.56904387052982]
  This paper proposes a new visual grounding task called multi-context visual grounding.<n>It aims to localize instances of interest across multiple images based on open-ended text prompts.<n>We benchmark over 20 state-of-the-art MLLMs and foundation models with potential multi-context visual grounding capabilities.
  arXiv  Detail & Related papers  (2024-10-16T07:52:57Z)
• MMSci: A Dataset for Graduate-Level Multi-Discipline Multimodal Scientific Understanding [59.41495657570397]
  We present a comprehensive dataset compiled from Nature Communications articles covering 72 scientific fields.<n>We evaluated 19 proprietary and open-source models on two benchmark tasks, figure captioning and multiple-choice, and conducted human expert annotation.<n>Fine-tuning Qwen2-VL-7B with our task-specific data achieved better performance than GPT-4o and even human experts in multiple-choice evaluations.
  arXiv  Detail & Related papers  (2024-07-06T00:40:53Z)
• LLMatDesign: Autonomous Materials Discovery with Large Language Models [5.481299708562135]
  New materials can have significant scientific and technological implications. Recent advances in machine learning have enabled data-driven methods to rapidly screen or generate promising materials. We introduce LLMatDesign, a novel framework for interpretable materials design powered by large language models.
  arXiv  Detail & Related papers  (2024-06-19T02:35:02Z)
• Are LLMs Ready for Real-World Materials Discovery? [10.87312197950899]
  Large Language Models (LLMs) create exciting possibilities for powerful language processing tools to accelerate research in materials science. While LLMs have great potential to accelerate materials understanding and discovery, they currently fall short in being practical materials science tools. We show relevant failure cases of LLMs in materials science that reveal current limitations of LLMs related to comprehending and reasoning over complex, interconnected materials science knowledge.
  arXiv  Detail & Related papers  (2024-02-07T19:10:36Z)
• Multimodal Foundation Models for Material Property Prediction and Discovery [7.167520424757711]
  We introduce Multimodal Learning for Materials (MultiMat), which enables self-supervised multi-modality training of foundation models for materials.<n>We demonstrate MultiMat's potential using data from the Materials Project database on multiple axes.
  arXiv  Detail & Related papers  (2023-11-30T18:35:29Z)
• Reconstructing Materials Tetrahedron: Challenges in Materials Information Extraction [23.489721319567025]
  We discuss, quantify, and document challenges in automated information extraction from materials science literature. This information is spread in multiple formats, such as tables, text, and images, and with little or no uniformity in reporting style. We hope the present work inspires researchers to address the challenges in a coherent fashion, providing a fillip to IE towards developing a materials knowledge base.
  arXiv  Detail & Related papers  (2023-10-12T14:57:24Z)
• MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts [170.01089233942594]
  MathVista is a benchmark designed to combine challenges from diverse mathematical and visual tasks. The best-performing GPT-4V model achieves an overall accuracy of 49.9%, substantially outperforming Bard, the second-best performer, by 15.1%. GPT-4V still falls short of human performance by 10.4%, as it often struggles to understand complex figures and perform rigorous reasoning.
  arXiv  Detail & Related papers  (2023-10-03T17:57:24Z)
• MuMuQA: Multimedia Multi-Hop News Question Answering via Cross-Media Knowledge Extraction and Grounding [131.8797942031366]
  We present a new QA evaluation benchmark with 1,384 questions over news articles that require cross-media grounding of objects in images onto text. Specifically, the task involves multi-hop questions that require reasoning over image-caption pairs to identify the grounded visual object being referred to and then predicting a span from the news body text to answer the question. We introduce a novel multimedia data augmentation framework, based on cross-media knowledge extraction and synthetic question-answer generation, to automatically augment data that can provide weak supervision for this task.
  arXiv  Detail & Related papers  (2021-12-20T18:23:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.