Bridging the Simulation-to-Reality Gap in Electron Microscope Calibration via VAE-EM Estimation

• URL: http://arxiv.org/abs/2603.16549v2
• Date: Thu, 19 Mar 2026 11:07:39 GMT
• Title: Bridging the Simulation-to-Reality Gap in Electron Microscope Calibration via VAE-EM Estimation
• Abstract summary: We tackle the calibration problem for Scanning Transmission Electron Microscopes (STEM) by employing variational autoencoders (VAEs)<n>VAEs learn low-dimensional representations of images, whereas most existing methods only extract scalar values.<n>We then simultaneously estimate the model that maps calibration parameters to encoded representations and optimal calibration parameters using an expectation (EM) approach.
• Score: 1.4719924357068723
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Electron microscopy has enabled many scientific breakthroughs across multiple fields. A key challenge is the tuning of microscope parameters based on images to overcome optical aberrations that deteriorate image quality. This calibration problem is challenging due to the high-dimensional and noisy nature of the diagnostic images, and the fact that optimal parameters cannot be identified from a single image. We tackle the calibration problem for Scanning Transmission Electron Microscopes (STEM) by employing variational autoencoders (VAEs), trained on simulated data, to learn low-dimensional representations of images, whereas most existing methods extract only scalar values. We then simultaneously estimate the model that maps calibration parameters to encoded representations and the optimal calibration parameters using an expectation maximization (EM) approach. This joint estimation explicitly addresses the simulation-to-reality gap inherent in data-driven methods that train on simulated data from a digital twin. We leverage the known symmetry property of the optical system to establish global identifiability of the joint estimation problem, ensuring that a unique optimum exists. We demonstrate that our approach is substantially faster and more consistent than existing methods on a real STEM, achieving a 2x reduction in estimation error while requiring fewer observations. This represents a notable advance in automated STEM calibration and demonstrates the potential of VAEs for information compression in images. Beyond microscopy, the VAE-EM framework applies to inverse problems where simulated training data introduces a reality gap and where non-injective mappings would otherwise prevent unique solutions.

Related papers

• PLS-Calib: A Partial Least Squares Framework for Event Camera and Odometry Calibration under Ground Motion Constraints [27.568640844773338]
  Most existing calibration techniques rely on full 6-DoF motion to excite all degrees of freedom.<n>We present a novel rotation calibration framework named PLS-Calib.<n>We show significant improvements in calibration robustness and accuracy over state-of-the-art methods.
  arXiv  Detail & Related papers  (2026-08-04T08:07:07Z)
• Addressing Class Imbalance and Data Limitations in Advanced Node Semiconductor Defect Inspection: A Generative Approach for SEM Images [0.10555513406636088]
  We propose a method for generating synthetic semiconductor SEM images using a diffusion model within a limited data regime. In contrast to images generated through conventional simulation methods, SEM images generated through our proposed DL method closely resemble real SEM images, replicating their noise characteristics and surface roughness adaptively.
  arXiv  Detail & Related papers  (2024-07-14T22:25:05Z)
• Single-image camera calibration with model-free distortion correction [0.0]
  This paper proposes a method for estimating the complete set of calibration parameters from a single image of a planar speckle pattern covering the entire sensor. The correspondence between image points and physical points on the calibration target is obtained using Digital Image Correlation. At the end of the procedure, a dense and uniform model-free distortion map is obtained over the entire image.
  arXiv  Detail & Related papers  (2024-03-02T16:51:35Z)
• Optimizations of Autoencoders for Analysis and Classification of Microscopic In Situ Hybridization Images [68.8204255655161]
  We propose a deep-learning framework to detect and classify areas of microscopic images with similar levels of gene expression. The data we analyze requires an unsupervised learning model for which we employ a type of Artificial Neural Network - Deep Learning Autoencoders.
  arXiv  Detail & Related papers  (2023-04-19T13:45:28Z)
• Differentiable Electron Microscopy Simulation: Methods and Applications for Visualization [40.8023670606058]
  We propose a new microscopy simulation system that can depict atomistic models in a micrograph visual style. The system is scalable, able to represent simulation of electron microscopy of tens of viral particles and synthesizes the image faster than previous methods.
  arXiv  Detail & Related papers  (2022-05-08T12:39:04Z)
• Negligible effect of brain MRI data preprocessing for tumor segmentation [36.89606202543839]
  We conduct experiments on three publicly available datasets and evaluate the effect of different preprocessing steps in deep neural networks. Our results demonstrate that most popular standardization steps add no value to the network performance. We suggest that image intensity normalization approaches do not contribute to model accuracy because of the reduction of signal variance with image standardization.
  arXiv  Detail & Related papers  (2022-04-11T17:29:36Z)
• Image-to-Image Regression with Distribution-Free Uncertainty Quantification and Applications in Imaging [88.20869695803631]
  We show how to derive uncertainty intervals around each pixel that are guaranteed to contain the true value. We evaluate our procedure on three image-to-image regression tasks.
  arXiv  Detail & Related papers  (2022-02-10T18:59:56Z)
• Deep Learning Adapted Acceleration for Limited-view Photoacoustic Computed Tomography [1.8830359888767887]
  Photoacoustic computed tomography (PACT) uses unfocused large-area light to illuminate the target with ultrasound transducer array for PA signal detection. Limited-view issue could cause a low-quality image in PACT due to the limitation of geometric condition. A model-based method that combines the mathematical variational model with deep learning is proposed to speed up and regularize the unrolled procedure of reconstruction.
  arXiv  Detail & Related papers  (2021-11-08T02:05:58Z)
• A parameter refinement method for Ptychography based on Deep Learning concepts [55.41644538483948]
  coarse parametrisation in propagation distance, position errors and partial coherence frequently menaces the experiment viability. A modern Deep Learning framework is used to correct autonomously the setup incoherences, thus improving the quality of a ptychography reconstruction. We tested our system on both synthetic datasets and also on real data acquired at the TwinMic beamline of the Elettra synchrotron facility.
  arXiv  Detail & Related papers  (2021-05-18T10:15:17Z)
• Leveraging Spatial and Photometric Context for Calibrated Non-Lambertian Photometric Stereo [61.6260594326246]
  We introduce an efficient fully-convolutional architecture that can leverage both spatial and photometric context simultaneously. Using separable 4D convolutions and 2D heat-maps reduces the size and makes more efficient.
  arXiv  Detail & Related papers  (2021-03-22T18:06:58Z)
• Statistical control for spatio-temporal MEG/EEG source imaging with desparsified multi-task Lasso [102.84915019938413]
  Non-invasive techniques like magnetoencephalography (MEG) or electroencephalography (EEG) offer promise of non-invasive techniques. The problem of source localization, or source imaging, poses however a high-dimensional statistical inference challenge. We propose an ensemble of desparsified multi-task Lasso (ecd-MTLasso) to deal with this problem.
  arXiv  Detail & Related papers  (2020-09-29T21:17:16Z)
• Improved Slice-wise Tumour Detection in Brain MRIs by Computing Dissimilarities between Latent Representations [68.8204255655161]
  Anomaly detection for Magnetic Resonance Images (MRIs) can be solved with unsupervised methods. We have proposed a slice-wise semi-supervised method for tumour detection based on the computation of a dissimilarity function in the latent space of a Variational AutoEncoder. We show that by training the models on higher resolution images and by improving the quality of the reconstructions, we obtain results which are comparable with different baselines.
  arXiv  Detail & Related papers  (2020-07-24T14:02:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.