Unsupervised Reward-Driven Image Segmentation in Automated Scanning Transmission Electron Microscopy Experiments

• URL: http://arxiv.org/abs/2409.12462v2
• Date: Fri, 20 Sep 2024 13:44:35 GMT
• Title: Unsupervised Reward-Driven Image Segmentation in Automated Scanning Transmission Electron Microscopy Experiments
• Authors: Kamyar Barakati, Utkarsh Pratiush, Austin C. Houston, Gerd Duscher, Sergei V. Kalinin,
• Abstract summary: Automated experiments in scanning transmission electron microscopy (STEM) require rapid image segmentation to optimize data representation for human interpretation, decision-making, site-selective spectroscopies, and atomic manipulation. Here, we operationalize and benchmark a recently proposed reward-driven optimization workflow for on-the fly image analysis in STEM. This unsupervised approach is much more robust, as it does not rely on human labels and is fully explainable.
• Score: 0.22795086293129713
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Automated experiments in scanning transmission electron microscopy (STEM) require rapid image segmentation to optimize data representation for human interpretation, decision-making, site-selective spectroscopies, and atomic manipulation. Currently, segmentation tasks are typically performed using supervised machine learning methods, which require human-labeled data and are sensitive to out-of-distribution drift effects caused by changes in resolution, sampling, or beam shape. Here, we operationalize and benchmark a recently proposed reward-driven optimization workflow for on-the fly image analysis in STEM. This unsupervised approach is much more robust, as it does not rely on human labels and is fully explainable. The explanatory feedback can help the human to verify the decision making and potentially tune the model by selecting the position along the Pareto frontier of reward functions. We establish the timing and effectiveness of this method, demonstrating its capability for real-time performance in high-throughput and dynamic automated STEM experiments. The reward driven approach allows to construct explainable robust analysis workflows and can be generalized to a broad range of image analysis tasks in electron and scanning probe microscopy and chemical imaging.

Related papers

• Reward driven workflows for unsupervised explainable analysis of phases and ferroic variants from atomically resolved imaging data [14.907891992968361]
  We show that a reward-driven approach can be used to optimize key hyper parameters in unsupervised ML methods. This approach allows us to discover local descriptors that are best aligned with the specific physical behavior. We also extend the reward driven to disentangle structural factors of variation via variational autoencoder (VAE)
  arXiv  Detail & Related papers  (2024-11-19T16:18:20Z)
• Foundational Model for Electron Micrograph Analysis: Instruction-Tuning Small-Scale Language-and-Vision Assistant for Enterprise Adoption [0.0]
  We introduce a small-scale framework for analyzing semiconductor electron microscopy images (MAEMI) We generate a customized instruction-following dataset using large multimodal models on microscopic image analysis. We perform knowledge transfer from larger to smaller models through knowledge distillation, resulting in improved accuracy of smaller models on visual question answering tasks.
  arXiv  Detail & Related papers  (2024-08-23T17:42:11Z)
• Efficient Visual State Space Model for Image Deblurring [83.57239834238035]
  Convolutional neural networks (CNNs) and Vision Transformers (ViTs) have achieved excellent performance in image restoration. We propose a simple yet effective visual state space model (EVSSM) for image deblurring.
  arXiv  Detail & Related papers  (2024-05-23T09:13:36Z)
• Deep Domain Adaptation: A Sim2Real Neural Approach for Improving Eye-Tracking Systems [80.62854148838359]
  Eye image segmentation is a critical step in eye tracking that has great influence over the final gaze estimate. We use dimensionality-reduction techniques to measure the overlap between the target eye images and synthetic training data. Our methods result in robust, improved performance when tackling the discrepancy between simulation and real-world data samples.
  arXiv  Detail & Related papers  (2024-03-23T22:32:06Z)
• Learning and Controlling Silicon Dopant Transitions in Graphene using Scanning Transmission Electron Microscopy [58.51812955462815]
  We introduce a machine learning approach to determine the transition dynamics of silicon atoms on a single layer of carbon atoms. The data samples are processed and filtered to produce symbolic representations, which we use to train a neural network to predict transition probabilities. These learned transition dynamics are then leveraged to guide a single silicon atom throughout the lattice to pre-determined target destinations.
  arXiv  Detail & Related papers  (2023-11-21T21:51:00Z)
• Self-Supervised Neuron Segmentation with Multi-Agent Reinforcement Learning [53.00683059396803]
  Mask image model (MIM) has been widely used due to its simplicity and effectiveness in recovering original information from masked images. We propose a decision-based MIM that utilizes reinforcement learning (RL) to automatically search for optimal image masking ratio and masking strategy. Our approach has a significant advantage over alternative self-supervised methods on the task of neuron segmentation.
  arXiv  Detail & Related papers  (2023-10-06T10:40:46Z)
• AI-assisted Automated Workflow for Real-time X-ray Ptychography Data Analysis via Federated Resources [2.682578132719034]
  We present an end-to-end automated workflow that uses large-scale remote compute resources and an embedded GPU platform at the edge to enable AI/ML-accelerated real-time analysis of data collected for x-ray ptychography.
  arXiv  Detail & Related papers  (2023-04-09T19:11:04Z)
• The Preliminary Results on Analysis of TAIGA-IACT Images Using Convolutional Neural Networks [68.8204255655161]
  The aim of the work is to study the possibility of the machine learning application to solve the tasks set for TAIGA-IACT. The method of Convolutional Neural Networks (CNN) was applied to process and analyze Monte-Carlo events simulated with CORSIKA.
  arXiv  Detail & Related papers  (2021-12-19T15:17:20Z)
• A parameter refinement method for Ptychography based on Deep Learning concepts [55.41644538483948]
  coarse parametrisation in propagation distance, position errors and partial coherence frequently menaces the experiment viability. A modern Deep Learning framework is used to correct autonomously the setup incoherences, thus improving the quality of a ptychography reconstruction. We tested our system on both synthetic datasets and also on real data acquired at the TwinMic beamline of the Elettra synchrotron facility.
  arXiv  Detail & Related papers  (2021-05-18T10:15:17Z)
• Automated and Autonomous Experiment in Electron and Scanning Probe Microscopy [0.0]
  We aim to analyze the major pathways towards automated experiment (AE) in imaging methods with sequential image formation mechanisms. We argue that automated experiments should necessarily be discussed in a broader context of the general domain knowledge that both informs the experiment and is increased as the result of the experiment.
  arXiv  Detail & Related papers  (2021-03-22T20:24:41Z)
• Ensemble learning and iterative training (ELIT) machine learning: applications towards uncertainty quantification and automated experiment in atom-resolved microscopy [0.0]
  Deep learning has emerged as a technique of choice for rapid feature extraction across imaging disciplines. Here we explore the application of deep learning for feature extraction in atom-resolved electron microscopy. This approach both allows uncertainty into deep learning analysis and also enables automated experimental detection where retraining of network to compensate for out-of-distribution drift due to change in imaging conditions is substituted for a human operator or programmatic selection of networks from the ensemble.
  arXiv  Detail & Related papers  (2021-01-21T05:29:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.