The microscope is the mask: privileged views and labels from a cryo-ET forward model

• URL: http://arxiv.org/abs/2609.04325v1
• Date: Thu, 03 Sep 2026 18:00:18 GMT
• Title: The microscope is the mask: privileged views and labels from a cryo-ET forward model
• Abstract summary: We explore the use of simulated data for training a model for protein annotation in crowded cryo-electron tomography volumes.<n>We leverage the corruptions imposed by the forward model to generate domain-specific augmented paired views of the exact same scene.<n>We show that CARNIVAL outperforms a state-of-the-art model trained using a contrastive objective on simulated data.
• Score: 3.4415392202192137
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We explore the use of simulated data for training a model for protein annotation in crowded cryo-electron tomography volumes reconstructed from images collected at limited tilt angles and severely corrupted by the measurement operator. Firstly, we leverage the corruptions imposed by the forward model to generate domain-specific augmented paired views of the exact same scene for an invariance objective integrated into the LeJEPA self-supervised training framework. Secondly, we use additional information from the simulation pipeline such as the positions and identity of proteins in the simulated volumes to inform the architecture of the model and the loss function, so that semantic information is localised at protein positions in the resulting dense feature volume. The resulting model, CARNIVAL, is evaluated without finetuning on classification and detection tasks in real tomograms, using a benchmark dataset containing multiple protein types and two tomogram processing types. We show that CARNIVAL outperforms a state-of-the-art model trained using a contrastive objective on simulated data but without forward model-based paired views or privileged information.

Related papers

• Physics Informed Generative AI Enabling Labour Free Segmentation For Microscopy Analysis [3.3176565054468714]
  This paper introduces a novel framework for labour-free segmentation that successfully bridges the simulation-to-reality gap.<n>We employ a Cycle-Consistent Generative Adversarial Network (CycleGAN) for unpaired image-to-image translation.<n>A U-Net model, trained exclusively on this synthetic data, demonstrated remarkable generalisation when deployed on unseen experimental images.
  arXiv  Detail & Related papers  (2026-02-02T06:36:06Z)
• Quantification and Classification of Carbon Nanotubes in Electron Micrographs using Vision Foundation Models [1.8969168959157112]
  This work presents a unified framework leveraging vision foundation models to automate the quantification and classification of CNTs in electron microscopy images.<n>We introduce an interactive quantification tool built on the Segment Anything Model (SAM) that segments particles with near-perfect accuracy using minimal user input.<n>Second, we propose a novel classification pipeline that utilizes these segmentation masks to spatially constrain a DINOv2 vision transformer, extracting features exclusively from particle regions while suppressing background noise.<n> Evaluated on a dataset of 1,800 TEM images, this architecture achieves 95.5% accuracy in distinguishing between four different CNT morphologies, significantly outperforming the current baseline despite using a
  arXiv  Detail & Related papers  (2026-01-10T20:22:58Z)
• Machine-learning-enabled interpretation of tribological deformation patterns in large-scale MD data [0.0]
  Grain-orientation-colored computational tomograph pictures obtained from CuNi alloy simulations were first compressed through an autoencoder to a 32-dimensional global feature vector.<n>The reconstructed images retained the essential microstructural motifs: grain boundaries, stacking faults, twins, and partial lattice rotations, while omitting only the finest defects.<n>A CNN-MLP model to predict the dominant deformation pattern achieves a prediction accuracy of approximately 96% on validation data.
  arXiv  Detail & Related papers  (2025-12-05T15:39:13Z)
• Steering Masked Discrete Diffusion Models via Discrete Denoising Posterior Prediction [88.65168366064061]
  We introduce Discrete Denoising Posterior Prediction (DDPP), a novel framework that casts the task of steering pre-trained MDMs as a problem of probabilistic inference. Our framework leads to a family of three novel objectives that are all simulation-free, and thus scalable. We substantiate our designs via wet-lab validation, where we observe transient expression of reward-optimized protein sequences.
  arXiv  Detail & Related papers  (2024-10-10T17:18:30Z)
• A Plug-and-Play Method for Rare Human-Object Interactions Detection by Bridging Domain Gap [50.079224604394]
  We present a novel model-agnostic framework called textbfContext-textbfEnhanced textbfFeature textbfAment (CEFA) CEFA consists of a feature alignment module and a context enhancement module. Our method can serve as a plug-and-play module to improve the detection performance of HOI models on rare categories.
  arXiv  Detail & Related papers  (2024-07-31T08:42:48Z)
• Reinforcing Pre-trained Models Using Counterfactual Images [54.26310919385808]
  This paper proposes a novel framework to reinforce classification models using language-guided generated counterfactual images. We identify model weaknesses by testing the model using the counterfactual image dataset. We employ the counterfactual images as an augmented dataset to fine-tune and reinforce the classification model.
  arXiv  Detail & Related papers  (2024-06-19T08:07:14Z)
• Deep Domain Adaptation: A Sim2Real Neural Approach for Improving Eye-Tracking Systems [80.62854148838359]
  Eye image segmentation is a critical step in eye tracking that has great influence over the final gaze estimate. We use dimensionality-reduction techniques to measure the overlap between the target eye images and synthetic training data. Our methods result in robust, improved performance when tackling the discrepancy between simulation and real-world data samples.
  arXiv  Detail & Related papers  (2024-03-23T22:32:06Z)
• Optimizations of Autoencoders for Analysis and Classification of Microscopic In Situ Hybridization Images [68.8204255655161]
  We propose a deep-learning framework to detect and classify areas of microscopic images with similar levels of gene expression. The data we analyze requires an unsupervised learning model for which we employ a type of Artificial Neural Network - Deep Learning Autoencoders.
  arXiv  Detail & Related papers  (2023-04-19T13:45:28Z)
• Self-Supervised Training with Autoencoders for Visual Anomaly Detection [61.62861063776813]
  We focus on a specific use case in anomaly detection where the distribution of normal samples is supported by a lower-dimensional manifold. We adapt a self-supervised learning regime that exploits discriminative information during training but focuses on the submanifold of normal examples. We achieve a new state-of-the-art result on the MVTec AD dataset -- a challenging benchmark for visual anomaly detection in the manufacturing domain.
  arXiv  Detail & Related papers  (2022-06-23T14:16:30Z)
• Reconstructing Training Data from Diverse ML Models by Ensemble Inversion [8.414622657659168]
  Model Inversion (MI), in which an adversary abuses access to a trained Machine Learning (ML) model, has attracted increasing research attention. We propose an ensemble inversion technique that estimates the distribution of original training data by training a generator constrained by an ensemble of trained models. We achieve high quality results without any dataset and show how utilizing an auxiliary dataset that's similar to the presumed training data improves the results.
  arXiv  Detail & Related papers  (2021-11-05T18:59:01Z)
• Y-GAN: Learning Dual Data Representations for Efficient Anomaly Detection [0.0]
  We propose a novel reconstruction-based model for anomaly detection, called Y-GAN. The model consists of a Y-shaped auto-encoder and represents images in two separate latent spaces.
  arXiv  Detail & Related papers  (2021-09-28T20:17:04Z)
• Robust Finite Mixture Regression for Heterogeneous Targets [70.19798470463378]
  We propose an FMR model that finds sample clusters and jointly models multiple incomplete mixed-type targets simultaneously. We provide non-asymptotic oracle performance bounds for our model under a high-dimensional learning framework. The results show that our model can achieve state-of-the-art performance.
  arXiv  Detail & Related papers  (2020-10-12T03:27:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.