Revisiting foundation models for cell instance segmentation

• URL: http://arxiv.org/abs/2603.17845v1
• Date: Wed, 18 Mar 2026 15:34:42 GMT
• Title: Revisiting foundation models for cell instance segmentation
• Abstract summary: We evaluate foundation models for cell segmentation (CellPoseSAM, CellSAM, $$SAM) and for general-purpose segmentation (SAM, SAM2, SAM3) on a diverse set of microscopy datasets.<n>We introduce a new instance segmentation strategy called automatic prompt generation (APG) that can be used to further improve SAM-based microscopy foundation models.<n>Our work provides important lessons for adaptation strategies of SAM-style models to microscopy and provides a strategy for creating even more powerful microscopy foundation models.
• Score: 1.514139641301594
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Cell segmentation is a fundamental task in microscopy image analysis. Several foundation models for cell segmentation have been introduced, virtually all of them are extensions of Segment Anything Model (SAM), improving it for microscopy data. Recently, SAM2 and SAM3 have been published, further improving and extending the capabilities of general-purpose segmentation foundation models. Here, we comprehensively evaluate foundation models for cell segmentation (CellPoseSAM, CellSAM, $μ$SAM) and for general-purpose segmentation (SAM, SAM2, SAM3) on a diverse set of (light) microscopy datasets, for tasks including cell, nucleus and organoid segmentation. Furthermore, we introduce a new instance segmentation strategy called automatic prompt generation (APG) that can be used to further improve SAM-based microscopy foundation models. APG consistently improves segmentation results for $μ$SAM, which is used as the base model, and is competitive with the state-of-the-art model CellPoseSAM. Moreover, our work provides important lessons for adaptation strategies of SAM-style models to microscopy and provides a strategy for creating even more powerful microscopy foundation models. Our code is publicly available at https://github.com/computational-cell-analytics/micro-sam.

Related papers

• $μ$Match: Foundation Models for Semi-supervised Learning and Domain Adaptation in EM [1.149497648076115]
  Vision foundation models have substantially advanced computer vision, enabling state-of-the-art performance in zero- and few-shot settings.<n>We propose $$Match, a framework for semi-supervised learning and domain adaptation that leverages foundation models.<n>We implement state-of-the-art student-teacher-based methods and evaluate multiple foundation models on challenging EM tasks, including mitochondrion, nucleus, and neurite segmentation.
  arXiv  Detail & Related papers  (2026-06-19T17:07:16Z)
• Self-supervised Pretraining of Cell Segmentation Models [37.09823149373976]
  We propose DINOCell, a self-supervised framework for cell instance segmentation.<n>It adapts representations from DINOv2 to microscopy through continued self-supervised training on unlabeled cell images.<n>On the LIVECell benchmark, DINOCell achieves a SEG score of 0.784, improving by 10.42% over leading SAM-based models.
  arXiv  Detail & Related papers  (2026-04-12T12:29:39Z)
• CellPainTR: Generalizable Representation Learning for Cross-Dataset Cell Painting Analysis [51.56484100374058]
  We introduce CellPainTR, a Transformer-based architecture designed to learn foundational representations of cellular morphology.<n>Our work represents a significant step towards creating truly foundational models for image-based profiling, enabling more reliable and scalable cross-study biological analysis.
  arXiv  Detail & Related papers  (2025-09-02T03:30:07Z)
• Fine-grained Multi-class Nuclei Segmentation with Molecular-empowered All-in-SAM Model [15.622506758735476]
  We propose the molecular-empowered All-in-SAM Model to advance computational pathology.<n>This model incorporates a full-stack approach, focusing on: (1) annotation-engaging lay annotators through molecular-empowered learning to reduce the need for detailed pixel-level annotations, and (2) learning-adapting the SAM model to emphasize specific semantics.
  arXiv  Detail & Related papers  (2025-08-21T17:49:21Z)
• Segment Anything for Histopathology [2.6579756198224347]
  Vision foundation models (VFMs) offer a more robust alternative for automatic and interactive segmentation.<n>We introduce PathoSAM, a VFM for nucleus segmentation based on training SAM on a diverse dataset.<n>Our models are open-source and compatible with popular tools for data annotation.
  arXiv  Detail & Related papers  (2025-02-01T11:59:04Z)
• Adapting Segment Anything Model for Unseen Object Instance Segmentation [70.60171342436092]
  Unseen Object Instance (UOIS) is crucial for autonomous robots operating in unstructured environments. We propose UOIS-SAM, a data-efficient solution for the UOIS task. UOIS-SAM integrates two key components: (i) a Heatmap-based Prompt Generator (HPG) to generate class-agnostic point prompts with precise foreground prediction, and (ii) a Hierarchical Discrimination Network (HDNet) that adapts SAM's mask decoder.
  arXiv  Detail & Related papers  (2024-09-23T19:05:50Z)
• Moving Object Segmentation: All You Need Is SAM (and Flow) [82.78026782967959]
  We investigate two models for combining SAM with optical flow that harness the segmentation power of SAM with the ability of flow to discover and group moving objects. In the first model, we adapt SAM to take optical flow, rather than RGB, as an input. In the second, SAM takes RGB as an input, and flow is used as a segmentation prompt. These surprisingly simple methods, without any further modifications, outperform all previous approaches by a considerable margin in both single and multi-object benchmarks.
  arXiv  Detail & Related papers  (2024-04-18T17:59:53Z)
• Mixed Models with Multiple Instance Learning [51.440557223100164]
  We introduce MixMIL, a framework integrating Generalized Linear Mixed Models (GLMM) and Multiple Instance Learning (MIL) Our empirical results reveal that MixMIL outperforms existing MIL models in single-cell datasets.
  arXiv  Detail & Related papers  (2023-11-04T16:42:42Z)
• SAM-CLIP: Merging Vision Foundation Models towards Semantic and Spatial Understanding [40.40630116715132]
  The landscape of publicly available vision foundation models (VFMs) is expanding rapidly. We introduce a simple recipe to efficiently merge VFMs into a unified model that absorbs their expertise. By applying our method to SAM and CLIP, we obtain SAM-CLIP: a unified model that combines the capabilities of SAM and CLIP into a single vision transformer.
  arXiv  Detail & Related papers  (2023-10-23T19:21:57Z)
• Multi-stream Cell Segmentation with Low-level Cues for Multi-modality Images [66.79688768141814]
  We develop an automatic cell classification pipeline to label microscopy images. We then train a classification model based on the category labels. We deploy two types of segmentation models to segment cells with roundish and irregular shapes.
  arXiv  Detail & Related papers  (2023-10-22T08:11:08Z)
• nnSAM: Plug-and-play Segment Anything Model Improves nnUNet Performance [12.169801149021566]
  The Segment Anything Model (SAM) has emerged as a versatile tool for image segmentation without specific domain training. Traditional models like nnUNet perform automatic segmentation during inference but need extensive domain-specific training. We propose nnSAM, integrating SAM's robust feature extraction with nnUNet's automatic configuration to enhance segmentation accuracy on small datasets.
  arXiv  Detail & Related papers  (2023-09-29T04:26:25Z)
• Semantic-SAM: Segment and Recognize Anything at Any Granularity [83.64686655044765]
  We introduce Semantic-SAM, a universal image segmentation model to enable segment and recognize anything at any desired granularity. We consolidate multiple datasets across three granularities and introduce decoupled classification for objects and parts. For the multi-granularity capability, we propose a multi-choice learning scheme during training, enabling each click to generate masks at multiple levels.
  arXiv  Detail & Related papers  (2023-07-10T17:59:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.