Zero-shot Shape Classification of Nanoparticles in SEM Images using Vision Foundation Models

• URL: http://arxiv.org/abs/2508.03235v1
• Date: Tue, 05 Aug 2025 09:03:56 GMT
• Title: Zero-shot Shape Classification of Nanoparticles in SEM Images using Vision Foundation Models
• Authors: Freida Barnatan, Emunah Goldstein, Einav Kalimian, Orchen Madar, Avi Huri, David Zitoun, Ya'akov Mandelbaum, Moshe Amitay,
• Abstract summary: Conventional deep learning methods for shape classification require extensive labeled datasets and computationally demanding training.<n>In this study, we introduce a zero-shot classification pipeline that leverages two vision foundation models.<n>We achieve high-precision shape classification across three morphologically diverse nanoparticle datasets.
• Score: 0.9466841964978984
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Accurate and efficient characterization of nanoparticle morphology in Scanning Electron Microscopy (SEM) images is critical for ensuring product quality in nanomaterial synthesis and accelerating development. However, conventional deep learning methods for shape classification require extensive labeled datasets and computationally demanding training, limiting their accessibility to the typical nanoparticle practitioner in research and industrial settings. In this study, we introduce a zero-shot classification pipeline that leverages two vision foundation models: the Segment Anything Model (SAM) for object segmentation and DINOv2 for feature embedding. By combining these models with a lightweight classifier, we achieve high-precision shape classification across three morphologically diverse nanoparticle datasets - without the need for extensive parameter fine-tuning. Our methodology outperforms a fine-tuned YOLOv11 and ChatGPT o4-mini-high baselines, demonstrating robustness to small datasets, subtle morphological variations, and domain shifts from natural to scientific imaging. Quantitative clustering metrics on PCA plots of the DINOv2 features are discussed as a means of assessing the progress of the chemical synthesis. This work highlights the potential of foundation models to advance automated microscopy image analysis, offering an alternative to traditional deep learning pipelines in nanoparticle research which is both more efficient and more accessible to the user.

Related papers

• F-ANcGAN: An Attention-Enhanced Cycle Consistent Generative Adversarial Architecture for Synthetic Image Generation of Nanoparticles [3.124884279860061]
  We introduce F-ANcGAN, an attention-enhanced cycle consistent generative adversarial system that can be trained using a limited number of data samples.<n>Our model uses a Style U-Net generator and a U-Net segmentation network equipped with self-attention to capture structural relationships.
  arXiv  Detail & Related papers  (2025-05-23T17:02:22Z)
• DiffRenderGAN: Addressing Training Data Scarcity in Deep Segmentation Networks for Quantitative Nanomaterial Analysis through Differentiable Rendering and Generative Modelling [0.1135917885955104]
  Deep learning segmentation networks enable automated insights and replace subjective methods with precise quantitative analysis.<n>We introduce DiffRenderGAN, a novel generative model designed to produce annotated synthetic data.<n>This approach reduces the need for manual intervention and enhances segmentation performance compared to existing synthetic data methods.
  arXiv  Detail & Related papers  (2025-02-13T16:41:44Z)
• MaskTerial: A Foundation Model for Automated 2D Material Flake Detection [48.73213960205105]
  We present a deep learning model, called MaskTerial, that uses an instance segmentation network to reliably identify 2D material flakes.<n>The model is extensively pre-trained using a synthetic data generator, that generates realistic microscopy images from unlabeled data.<n>We demonstrate significant improvements over existing techniques in the detection of low-contrast materials such as hexagonal boron nitride.
  arXiv  Detail & Related papers  (2024-12-12T15:01:39Z)
• Improving Molecular Modeling with Geometric GNNs: an Empirical Study [56.52346265722167]
  This paper focuses on the impact of different canonicalization methods, (2) graph creation strategies, and (3) auxiliary tasks, on performance, scalability and symmetry enforcement. Our findings and insights aim to guide researchers in selecting optimal modeling components for molecular modeling tasks.
  arXiv  Detail & Related papers  (2024-07-11T09:04:12Z)
• Benchmark on Drug Target Interaction Modeling from a Structure Perspective [48.60648369785105]
  Drug-target interaction prediction is crucial to drug discovery and design. Recent methods, such as those based on graph neural networks (GNNs) and Transformers, demonstrate exceptional performance across various datasets. We conduct a comprehensive survey and benchmark for drug-target interaction modeling from a structure perspective, via integrating tens of explicit (i.e., GNN-based) and implicit (i.e., Transformer-based) structure learning algorithms.
  arXiv  Detail & Related papers  (2024-07-04T16:56:59Z)
• Automated Classification of Nanoparticles with Various Ultrastructures and Sizes [0.6927055673104933]
  We present a deep-learning based method for nanoparticles measurement and classification trained from a small data set of scanning transmission electron microscopy images. Our approach is comprised of two stages: localization, i.e., detection of nanoparticles, and classification, i.e., categorization of their ultrastructure. We show how the generation of synthetic images, either using image processing or using various image generation neural networks, can be used to improve the results in both stages.
  arXiv  Detail & Related papers  (2022-07-28T11:31:43Z)
• Domain Adaptive Nuclei Instance Segmentation and Classification via Category-aware Feature Alignment and Pseudo-labelling [65.40672505658213]
  We propose a novel deep neural network, namely Category-Aware feature alignment and Pseudo-Labelling Network (CAPL-Net) for UDA nuclei instance segmentation and classification. Our approach outperforms state-of-the-art UDA methods with a remarkable margin.
  arXiv  Detail & Related papers  (2022-07-04T07:05:06Z)
• Deep-learning-based prediction of nanoparticle phase transitions during in situ transmission electron microscopy [3.613625739845355]
  We train deep learning models to predict a sequence of future video frames based on the input of a sequence of previous frames. This capability provides insight into size dependent structural changes in Au nanoparticles under dynamic reaction condition. It may be possible to anticipate the next steps of a chemical reaction for emerging automated experimentation platforms.
  arXiv  Detail & Related papers  (2022-05-23T15:50:24Z)
• Synthetic Image Rendering Solves Annotation Problem in Deep Learning Nanoparticle Segmentation [5.927116192179681]
  We show that using a rendering software allows to generate realistic, synthetic training data to train a state-of-the art deep neural network. We derive a segmentation accuracy that is comparable to man-made annotations for toxicologically relevant metal-oxide nanoparticles ensembles.
  arXiv  Detail & Related papers  (2020-11-20T17:05:36Z)
• Towards an Automatic Analysis of CHO-K1 Suspension Growth in Microfluidic Single-cell Cultivation [63.94623495501023]
  We propose a novel Machine Learning architecture, which allows us to infuse a neural deep network with human-powered abstraction on the level of data. Specifically, we train a generative model simultaneously on natural and synthetic data, so that it learns a shared representation, from which a target variable, such as the cell count, can be reliably estimated.
  arXiv  Detail & Related papers  (2020-10-20T08:36:51Z)
• Modeling Shared Responses in Neuroimaging Studies through MultiView ICA [94.31804763196116]
  Group studies involving large cohorts of subjects are important to draw general conclusions about brain functional organization. We propose a novel MultiView Independent Component Analysis model for group studies, where data from each subject are modeled as a linear combination of shared independent sources plus noise. We demonstrate the usefulness of our approach first on fMRI data, where our model demonstrates improved sensitivity in identifying common sources among subjects.
  arXiv  Detail & Related papers  (2020-06-11T17:29:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.