AutoMat: Enabling Automated Crystal Structure Reconstruction from Microscopy via Agentic Tool Use

• URL: http://arxiv.org/abs/2505.12650v1
• Date: Mon, 19 May 2025 03:04:50 GMT
• Title: AutoMat: Enabling Automated Crystal Structure Reconstruction from Microscopy via Agentic Tool Use
• Authors: Yaotian Yang, Yiwen Tang, Yizhe Chen, Xiao Chen, Jiangjie Qiu, Hao Xiong, Haoyu Yin, Zhiyao Luo, Yifei Zhang, Sijia Tao, Wentao Li, Qinghua Zhang, Yuqiang Li, Wanli Ouyang, Bin Zhao, Xiaonan Wang, Fei Wei,
• Abstract summary: We introduce AutoMat, an end-to-end, agent-assisted pipeline that automatically transforms scanning transmission electron microscopy images into atomic crystal structures.<n>AutoMat combines pattern-adaptive denoising, physics-guided template retrieval, symmetry-aware atomic reconstruction, fast relaxation and property prediction via MatterSim.<n>In large-scale experiments over 450 structure samples, AutoMat substantially outperforms existing multimodal large language models and tools.
• Score: 43.82226218242389
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Machine learning-based interatomic potentials and force fields depend critically on accurate atomic structures, yet such data are scarce due to the limited availability of experimentally resolved crystals. Although atomic-resolution electron microscopy offers a potential source of structural data, converting these images into simulation-ready formats remains labor-intensive and error-prone, creating a bottleneck for model training and validation. We introduce AutoMat, an end-to-end, agent-assisted pipeline that automatically transforms scanning transmission electron microscopy (STEM) images into atomic crystal structures and predicts their physical properties. AutoMat combines pattern-adaptive denoising, physics-guided template retrieval, symmetry-aware atomic reconstruction, fast relaxation and property prediction via MatterSim, and coordinated orchestration across all stages. We propose the first dedicated STEM2Mat-Bench for this task and evaluate performance using lattice RMSD, formation energy MAE, and structure-matching success rate. By orchestrating external tool calls, AutoMat enables a text-only LLM to outperform vision-language models in this domain, achieving closed-loop reasoning throughout the pipeline. In large-scale experiments over 450 structure samples, AutoMat substantially outperforms existing multimodal large language models and tools. These results validate both AutoMat and STEM2Mat-Bench, marking a key step toward bridging microscopy and atomistic simulation in materials science.The code and dataset are publicly available at https://github.com/yyt-2378/AutoMat and https://huggingface.co/datasets/yaotianvector/STEM2Mat.

Related papers

• evoxels: A differentiable physics framework for voxel-based microstructure simulations [41.94295877935867]
  Differentiable physics framework evoxels is based on a fully Pythonic, unified voxel-based approach that integrates segmented 3D microscopy data, physical simulations, inverse modeling, and machine learning.
  arXiv  Detail & Related papers  (2025-07-29T12:29:15Z)
• Materials Learning Algorithms (MALA): Scalable Machine Learning for Electronic Structure Calculations in Large-Scale Atomistic Simulations [2.04071520659173]
  We present the Materials Learning Algorithms (MALA) package, a scalable machine learning framework suitable for large-scale atomistic simulations.<n>MALA models efficiently predict key electronic observables, including local density of states, electronic density, density of states, and total energy.<n>We demonstrate MALA's capabilities with examples including boron clusters, aluminum across its solid-liquid phase boundary, and predicting the electronic structure of a stacking fault in a large beryllium slab.
  arXiv  Detail & Related papers  (2024-11-29T11:10:29Z)
• OminiControl: Minimal and Universal Control for Diffusion Transformer [68.3243031301164]
  We present OminiControl, a novel approach that rethinks how image conditions are integrated into Diffusion Transformer (DiT) architectures.<n>OminiControl addresses these limitations through three key innovations.
  arXiv  Detail & Related papers  (2024-11-22T17:55:15Z)
• Embedding an ANN-Based Crystal Plasticity Model into the Finite Element Framework using an ABAQUS User-Material Subroutine [3.8090193872025226]
  This manuscript presents a practical method for incorporating trained Neural Networks (NNs) into the Finite Element (FE) framework using a user material (UMAT) subroutine. The work exemplifies crystal plasticity, a complex inelastic non-linear path-dependent material response, with a wide range of applications in ABAQUS UMAT.
  arXiv  Detail & Related papers  (2024-09-25T10:47:18Z)
• Fine-Tuned Language Models Generate Stable Inorganic Materials as Text [53.81190146434045]
  Fine-tuning large language models on text-encoded atomistic data is simple to implement yet reliable.<n>We show that our strongest model can generate materials predicted to be metastable at about twice the rate of CDVAE.<n>Because of text prompting's inherent flexibility, our models can simultaneously be used for unconditional generation of stable material.
  arXiv  Detail & Related papers  (2024-02-06T20:35:28Z)
• A robust synthetic data generation framework for machine learning in High-Resolution Transmission Electron Microscopy (HRTEM) [1.0923877073891446]
  Construction Zone is a Python package for rapidly generating complex nanoscale atomic structures. We develop an end-to-end workflow for creating large simulated databases for training neural networks. Using our results, we are able to achieve state-of-the-art segmentation performance on experimental HRTEM images of nanoparticles.
  arXiv  Detail & Related papers  (2023-09-12T10:44:15Z)
• FakET: Simulating Cryo-Electron Tomograms with Neural Style Transfer [4.0998481751764]
  We introduce FakET, capable of simulating the forward operator of any cryo transmission electron microscope.<n>It can be used to adapt a synthetic training data set according to reference data producing high-quality simulated micrographs or tilt-series.<n>Remarkably, our technique matches the performance, boosts data generation speed 750 times, uses 33 times less memory, and scales well to typical transmission electron microscope detector sizes.
  arXiv  Detail & Related papers  (2023-04-04T17:59:09Z)
• Multitask AET with Orthogonal Tangent Regularity for Dark Object Detection [84.52197307286681]
  We propose a novel multitask auto encoding transformation (MAET) model to enhance object detection in a dark environment. In a self-supervision manner, the MAET learns the intrinsic visual structure by encoding and decoding the realistic illumination-degrading transformation. We have achieved the state-of-the-art performance using synthetic and real-world datasets.
  arXiv  Detail & Related papers  (2022-05-06T16:27:14Z)
• SOLIS -- The MLOps journey from data acquisition to actionable insights [62.997667081978825]
  In this paper we present a unified deployment pipeline and freedom-to-operate approach that supports all requirements while using basic cross-platform tensor framework and script language engines. This approach however does not supply the needed procedures and pipelines for the actual deployment of machine learning capabilities in real production grade systems.
  arXiv  Detail & Related papers  (2021-12-22T14:45:37Z)
• A parameter refinement method for Ptychography based on Deep Learning concepts [55.41644538483948]
  coarse parametrisation in propagation distance, position errors and partial coherence frequently menaces the experiment viability. A modern Deep Learning framework is used to correct autonomously the setup incoherences, thus improving the quality of a ptychography reconstruction. We tested our system on both synthetic datasets and also on real data acquired at the TwinMic beamline of the Elettra synchrotron facility.
  arXiv  Detail & Related papers  (2021-05-18T10:15:17Z)
• AtomAI: A Deep Learning Framework for Analysis of Image and Spectroscopy Data in (Scanning) Transmission Electron Microscopy and Beyond [0.0]
  AtomAI is an open-source software package bridging instrument-specific Python libraries, deep learning, and simulation tools into a single ecosystem. AtomAI allows direct applications of the deep convolutional neural networks for atomic and mesoscopic image segmentation. AtomAI provides utilities for mapping structure-property relationships via im2spec and spec2im type of encoder-decoder models.
  arXiv  Detail & Related papers  (2021-05-16T17:44:59Z)
• Meta-Sim2: Unsupervised Learning of Scene Structure for Synthetic Data Generation [88.04759848307687]
  In Meta-Sim2, we aim to learn the scene structure in addition to parameters, which is a challenging problem due to its discrete nature. We use Reinforcement Learning to train our model, and design a feature space divergence between our synthesized and target images that is key to successful training. We also show that this leads to downstream improvement in the performance of an object detector trained on our generated dataset as opposed to other baseline simulation methods.
  arXiv  Detail & Related papers  (2020-08-20T17:28:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.