A Mixture of Experts Foundation Model for Scanning Electron Microscopy Image Analysis

• URL: http://arxiv.org/abs/2604.05960v1
• Date: Tue, 07 Apr 2026 14:52:43 GMT
• Title: A Mixture of Experts Foundation Model for Scanning Electron Microscopy Image Analysis
• Abstract summary: We introduce the first foundation model for SEM images, pretrained on a large corpus of multi-instrument, multi-condition scientific micrographs.<n>We focus on defocus-to-focus image translation-an essential yet underexplored challenge in automated microscopy pipelines.
• Score: 16.071217652115457
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Scanning Electron Microscopy (SEM) is indispensable in modern materials science, enabling high-resolution imaging across a wide range of structural, chemical, and functional investigations. However, SEM imaging remains constrained by task-specific models and labor-intensive acquisition processes that limit its scalability across diverse applications. Here, we introduce the first foundation model for SEM images, pretrained on a large corpus of multi-instrument, multi-condition scientific micrographs, enabling generalization across diverse material systems and imaging conditions. Leveraging a self-supervised transformer architecture, our model learns rich and transferable representations that can be fine-tuned or adapted to a wide range of downstream tasks. As a compelling demonstration, we focus on defocus-to-focus image translation-an essential yet underexplored challenge in automated microscopy pipelines. Our method not only restores focused detail from defocused inputs without paired supervision but also outperforms state-of-the-art techniques across multiple evaluation metrics. This work lays the groundwork for a new class of adaptable SEM models, accelerating materials discovery by bridging foundational representation learning with real-world imaging needs.

Related papers

• scMIR: a vision-language foundation model for single-cell light microscopy image representation [9.935076530517575]
  We propose scMIR, a vision-language foundation model for single-cell light microscopy image representation.<n> scMIR combines self-supervised image reconstruction with text-guided cross-modal alignment.<n> scMIR shows a strong generalization ability across various tasks without requiring task-specific fine-tuning.
  arXiv  Detail & Related papers  (2026-07-21T01:59:07Z)
• More Images, More Problems? A Controlled Analysis of VLM Failure Modes [80.64323947730905]
  Large Vision Language Models (LVLMs) have demonstrated remarkable capabilities, yet their proficiency in understanding and reasoning over multiple images remains largely unexplored.<n>We introduce MIMIC, a new benchmark designed to rigorously evaluate the multi-image capabilities of LVLMs.
  arXiv  Detail & Related papers  (2026-01-12T18:45:13Z)
• Disentangled representations of microscopy images [0.9849635250118911]
  This work proposes a Disentangled Representation Learning (DRL) methodology to enhance model interpretability for microscopy image classification.<n>We show how a DRL framework, based on transferring a representation learnt from synthetic data, can provide a good trade-off between accuracy and interpretability in this domain.
  arXiv  Detail & Related papers  (2025-06-25T17:44:37Z)
• Uni-AIMS: AI-Powered Microscopy Image Analysis [28.24402780080126]
  We develop a data engine that generates high-quality annotated datasets.<n>We propose a segmentation model capable of robustly detecting both small and large objects.<n>Our solution supports the precise automatic recognition of image scale bars.
  arXiv  Detail & Related papers  (2025-05-11T09:35:53Z)
• IDArb: Intrinsic Decomposition for Arbitrary Number of Input Views and Illuminations [64.07859467542664]
  Capturing geometric and material information from images remains a fundamental challenge in computer vision and graphics.<n>Traditional optimization-based methods often require hours of computational time to reconstruct geometry, material properties, and environmental lighting from dense multi-view inputs.<n>We introduce IDArb, a diffusion-based model designed to perform intrinsic decomposition on an arbitrary number of images under varying illuminations.
  arXiv  Detail & Related papers  (2024-12-16T18:52:56Z)
• Deep learning assisted high resolution microscopy image processing for phase segmentation in functional composite materials [8.01747783558987]
  This work proposes a novel workflow for detecting components and phase segmentation from raw high resolution transmission electron microscopy (TEM) images using a trained U-Net segmentation model.<n>The developed model can expedite the detection of components and phase segmentation, diminishing the temporal and cognitive demands associated with scrutinizing an extensive array of TEM images.
  arXiv  Detail & Related papers  (2024-10-02T18:22:04Z)
• Multi-Modal Instruction-Tuning Small-Scale Language-and-Vision Assistant for Semiconductor Electron Micrograph Analysis [0.0]
  We present a novel framework for analyzing and interpreting electron microscopy images in semiconductor manufacturing. The framework employs a unique teacher-student approach, leveraging pre-trained multimodal large language models.
  arXiv  Detail & Related papers  (2024-08-27T15:50:04Z)
• Foundational Model for Electron Micrograph Analysis: Instruction-Tuning Small-Scale Language-and-Vision Assistant for Enterprise Adoption [0.0]
  We introduce a small-scale framework for analyzing semiconductor electron microscopy images (MAEMI) We generate a customized instruction-following dataset using large multimodal models on microscopic image analysis. We perform knowledge transfer from larger to smaller models through knowledge distillation, resulting in improved accuracy of smaller models on visual question answering tasks.
  arXiv  Detail & Related papers  (2024-08-23T17:42:11Z)
• MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training [103.72844619581811]
  We build performant Multimodal Large Language Models (MLLMs) In particular, we study the importance of various architecture components and data choices. We demonstrate that for large-scale multimodal pre-training using a careful mix of image-caption, interleaved image-text, and text-only data.
  arXiv  Detail & Related papers  (2024-03-14T17:51:32Z)
• Learning Multiscale Consistency for Self-supervised Electron Microscopy Instance Segmentation [48.267001230607306]
  We propose a pretraining framework that enhances multiscale consistency in EM volumes. Our approach leverages a Siamese network architecture, integrating strong and weak data augmentations. It effectively captures voxel and feature consistency, showing promise for learning transferable representations for EM analysis.
  arXiv  Detail & Related papers  (2023-08-19T05:49:13Z)
• Domain Generalization for Mammographic Image Analysis with Contrastive Learning [62.25104935889111]
  The training of an efficacious deep learning model requires large data with diverse styles and qualities. A novel contrastive learning is developed to equip the deep learning models with better style generalization capability. The proposed method has been evaluated extensively and rigorously with mammograms from various vendor style domains and several public datasets.
  arXiv  Detail & Related papers  (2023-04-20T11:40:21Z)
• Physics Embedded Machine Learning for Electromagnetic Data Imaging [83.27424953663986]
  Electromagnetic (EM) imaging is widely applied in sensing for security, biomedicine, geophysics, and various industries. It is an ill-posed inverse problem whose solution is usually computationally expensive. Machine learning (ML) techniques and especially deep learning (DL) show potential in fast and accurate imaging. This article surveys various schemes to incorporate physics in learning-based EM imaging.
  arXiv  Detail & Related papers  (2022-07-26T02:10:15Z)
• Image-specific Convolutional Kernel Modulation for Single Image Super-resolution [85.09413241502209]
  In this issue, we propose a novel image-specific convolutional modulation kernel (IKM) We exploit the global contextual information of image or feature to generate an attention weight for adaptively modulating the convolutional kernels. Experiments on single image super-resolution show that the proposed methods achieve superior performances over state-of-the-art methods.
  arXiv  Detail & Related papers  (2021-11-16T11:05:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.