scMIR: a vision-language foundation model for single-cell light microscopy image representation

• URL: http://arxiv.org/abs/2607.22712v2
• Date: Tue, 28 Jul 2026 02:14:54 GMT
• Title: scMIR: a vision-language foundation model for single-cell light microscopy image representation
• Abstract summary: We propose scMIR, a vision-language foundation model for single-cell light microscopy image representation.<n> scMIR combines self-supervised image reconstruction with text-guided cross-modal alignment.<n> scMIR shows a strong generalization ability across various tasks without requiring task-specific fine-tuning.
• Score: 9.935076530517575
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Single-cell light microscopy images have become an important data source for characterizing cell phenotypes, but their complexity and heterogeneity pose challenges to high-throughput automated analysis. Existing representation learning methods mostly rely on task-oriented modeling, which is limited by specific datasets and predefined tasks, making them difficult to generalize across different cell types and microscopy modalities, and experimental conditions. Although general-purpose methods have improved the generalization ability of image representation in recent years, their limited utilization of experimental background and biological context information still poses challenges in complex phenotypic analysis. Here, we propose scMIR, a vision-language foundation model for single-cell light microscopy image representation. By synergistically combining self-supervised image reconstruction with text-guided cross-modal alignment, scMIR can simultaneously encode morphological and biological semantic information in a unified representation space. scMIR is pre-trained on 207,957 image-text pairs, covering various cell types, microscopy modalities, and perturbation conditions. scMIR outperforms existing general models and task-oriented methods as systematically evaluated on various complex tasks using 16 benchmark datasets, including cell classification, clustering, phenotype inference, and batch effect correction tasks. Furthermore, scMIR shows a strong generalization ability across various tasks without requiring task-specific fine-tuning. With its unique advantages, we envision scMIR may promote the standardization and automation of high-throughput phenotyping workflows through supporting various downstream analysis tasks.

Related papers

• A Mixture of Experts Foundation Model for Scanning Electron Microscopy Image Analysis [16.071217652115457]
  We introduce the first foundation model for SEM images, pretrained on a large corpus of multi-instrument, multi-condition scientific micrographs.<n>We focus on defocus-to-focus image translation-an essential yet underexplored challenge in automated microscopy pipelines.
  arXiv  Detail & Related papers  (2026-04-07T14:52:43Z)
• Multimodal Causal-Driven Representation Learning for Generalizable Medical Image Segmentation [56.52520416420957]
  We propose Multimodal Causal-Driven Representation Learning (MCDRL) to tackle domain generalization in medical image segmentation.<n>MCDRL consistently outperforms competing methods, yielding superior segmentation accuracy and exhibiting robust generalizability.
  arXiv  Detail & Related papers  (2025-08-07T03:41:41Z)
• Disentangled representations of microscopy images [0.9849635250118911]
  This work proposes a Disentangled Representation Learning (DRL) methodology to enhance model interpretability for microscopy image classification.<n>We show how a DRL framework, based on transferring a representation learnt from synthetic data, can provide a good trade-off between accuracy and interpretability in this domain.
  arXiv  Detail & Related papers  (2025-06-25T17:44:37Z)
• Self-supervised Representation Learning with Local Aggregation for Image-based Profiling [84.52554180480037]
  Image-based cell profiling aims to create informative representations of cell images.<n>Recent developments in non-contrastive Self-Supervised Learning have inspired this paper.<n>We introduce specialized data augmentation and representation post-processing methods tailored to cell images.
  arXiv  Detail & Related papers  (2025-06-17T07:25:57Z)
• Uni-AIMS: AI-Powered Microscopy Image Analysis [28.24402780080126]
  We develop a data engine that generates high-quality annotated datasets.<n>We propose a segmentation model capable of robustly detecting both small and large objects.<n>Our solution supports the precise automatic recognition of image scale bars.
  arXiv  Detail & Related papers  (2025-05-11T09:35:53Z)
• CytoFM: The first cytology foundation model [3.591868126270513]
  We introduce CytoFM, the first self-supervised foundation model for digital Cytology.<n>We pretrain CytoFM on a diverse collection of datasets to learn robust, transferable representations.<n>Our results demonstrate that CytoFM performs better on two out of three downstream tasks than existing foundation models pretrained on histopathology.
  arXiv  Detail & Related papers  (2025-04-18T01:37:50Z)
• Learning Multiscale Consistency for Self-supervised Electron Microscopy Instance Segmentation [48.267001230607306]
  We propose a pretraining framework that enhances multiscale consistency in EM volumes. Our approach leverages a Siamese network architecture, integrating strong and weak data augmentations. It effectively captures voxel and feature consistency, showing promise for learning transferable representations for EM analysis.
  arXiv  Detail & Related papers  (2023-08-19T05:49:13Z)
• Optimizations of Autoencoders for Analysis and Classification of Microscopic In Situ Hybridization Images [68.8204255655161]
  We propose a deep-learning framework to detect and classify areas of microscopic images with similar levels of gene expression. The data we analyze requires an unsupervised learning model for which we employ a type of Artificial Neural Network - Deep Learning Autoencoders.
  arXiv  Detail & Related papers  (2023-04-19T13:45:28Z)
• AMIGO: Sparse Multi-Modal Graph Transformer with Shared-Context Processing for Representation Learning of Giga-pixel Images [53.29794593104923]
  We present a novel concept of shared-context processing for whole slide histopathology images. AMIGO uses the celluar graph within the tissue to provide a single representation for a patient. We show that our model is strongly robust to missing information to an extent that it can achieve the same performance with as low as 20% of the data.
  arXiv  Detail & Related papers  (2023-03-01T23:37:45Z)
• HAMIL: Hierarchical Aggregation-Based Multi-Instance Learning for Microscopy Image Classification [4.566276053984716]
  Multi-instance learning is common for computer vision tasks, especially in biomedical image processing. In this study, we propose a hierarchical aggregation network for multi-instance learning, called HAMIL. The hierarchical aggregation protocol enables feature fusion in a defined order, and the simple convolutional aggregation units lead to an efficient and flexible architecture.
  arXiv  Detail & Related papers  (2021-03-17T16:34:08Z)
• MOGAN: Morphologic-structure-aware Generative Learning from a Single Image [59.59698650663925]
  Recently proposed generative models complete training based on only one image. We introduce a MOrphologic-structure-aware Generative Adversarial Network named MOGAN that produces random samples with diverse appearances. Our approach focuses on internal features including the maintenance of rational structures and variation on appearance.
  arXiv  Detail & Related papers  (2021-03-04T12:45:23Z)
• Towards an Automatic Analysis of CHO-K1 Suspension Growth in Microfluidic Single-cell Cultivation [63.94623495501023]
  We propose a novel Machine Learning architecture, which allows us to infuse a neural deep network with human-powered abstraction on the level of data. Specifically, we train a generative model simultaneously on natural and synthetic data, so that it learns a shared representation, from which a target variable, such as the cell count, can be reliably estimated.
  arXiv  Detail & Related papers  (2020-10-20T08:36:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.