Rethinking the Adaptation of Vision Foundation Models for Efficient Cell Segmentation

• URL: http://arxiv.org/abs/2606.21913v1
• Date: Sat, 20 Jun 2026 07:15:14 GMT
• Title: Rethinking the Adaptation of Vision Foundation Models for Efficient Cell Segmentation
• Abstract summary: EffiCell-Seg is a framework for highly efficient cell segmentation without re-training the visual encoder.<n>Our core insight is that pretrained VFMs intrinsically encode complementary structural priors.<n>EffiCell-Seg requires only 5M trainable parameters, over 130x fewer than fully fine-tuned VFM counterparts.
• Score: 38.881896184660164
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Cell segmentation is critical for computational pathology and biomedical discovery. While recent Vision Foundation Models (VFMs) have demonstrated remarkable universal feature representations, unlocking their full potential for cellular imaging is currently bottlenecked by resource-intensive adaptation paradigms. Existing methods typically rely on fine-tuning heavy visual encoders, leading to extensive computational overhead and a dependency on large-scale annotations. To address this, we propose the EffiCell-Seg framework for highly efficient cell segmentation without re-training the visual encoder. Our core insight is that pretrained VFMs intrinsically encode complementary structural priors: global saliency for localizing potential cells, and local morphological patterns for delineating cellular structures. To harness these priors, we devise a Cell Structure Prompt Encoder (CSP-Encoder) that synthesizes semantic-aware saliency and principal morphological features from frozen VFM representations into explicit structural prior maps. Moreover, we propose a Synergistic Mask Decoder (SM-Decoder) that enforces contextual consistency by jointly predicting geometric distance fields and semantic maps via mutual cross-guidance. Extensive experiments demonstrate that EffiCell-Seg outperforms state-of-the-art methods across diverse cell imaging modalities while requiring only ~5M trainable parameters, over 130x fewer than fully fine-tuned VFM counterparts. The code is available at https://github.com/xq141839/EffiCell-Seg.

Related papers

• DualGate-Net: A Prior-Gated Dual-Encoder Framework for Histopathology Cell Detection [7.024374684283387]
  We propose DualGate-Net, a prior-aware dual-encoder framework that combines a ConvNeXtV2-based local encoder and a SegFormer-based global encoder through a learnable prior-gated fusion mechanism.<n> Experiments on the OCELOT benchmark demonstrate consistent improvements, achieving macro F1-scores of 0.7722 on the validation set and 0.7345 on the test set.
  arXiv  Detail & Related papers  (2026-06-05T12:36:05Z)
• Vision Transformer-Conditioned UNet for Domain-Adaptive Semantic Segmentation [3.2625979527355784]
  We introduce ViTC-UNet, which conditions a UNet on frozen pre-trained ViT representations through learnable tokens and a two-way attention decoder.<n>This combines ViT global visual priors with the local inductive bias and high-resolution decoding capacity of UNets.<n>ViTC-UNet outperforms baseline results in semantic segmentation tasks across MRI and CT modalities.
  arXiv  Detail & Related papers  (2026-05-12T11:56:46Z)
• BGG: Bridging the Geometric Gap between Cross-View images by Vision Foundation Model Adaptation for Geo-Localization [50.74663742490919]
  Cross-View Geo-Localization (CVGL) aims to acquire the geolocation of images by image retrieval.<n>This paper proposes a parameter-efficient framework for bridging the geometric gap across images based on the vision foundation model (VFM)<n>It mainly contains a Multi-granularity Feature Enhancement Adapter (MFEA) and a Frequency-Aware Structural Aggregation (FASA) module.
  arXiv  Detail & Related papers  (2026-05-11T10:46:33Z)
• TGC-Net: A Structure-Aware and Semantically-Aligned Framework for Text-Guided Medical Image Segmentation [56.09179939570486]
  We propose TGC-Net, a CLIP-based framework focusing on parameter-efficient, task-specific adaptations.<n>TGC-Net achieves state-of-the-art performance with substantially fewer trainable parameters, including notable Dice gains on challenging benchmarks.
  arXiv  Detail & Related papers  (2025-12-24T12:06:26Z)
• LapFM: A Laparoscopic Segmentation Foundation Model via Hierarchical Concept Evolving Pre-training [39.51756689308245]
  We present LapFM, a foundation model designed to evolve robust segmentation capabilities from massive unlabeled surgical images.<n>We propose a Confidence-driven Evolving Labeling that iteratively generates and filters pseudo-labels based on hierarchical consistency.<n>This process yields LapBench-114K, a large-scale benchmark comprising 114K image-mask pairs.
  arXiv  Detail & Related papers  (2025-12-09T10:09:31Z)
• CellPainTR: Generalizable Representation Learning for Cross-Dataset Cell Painting Analysis [51.56484100374058]
  We introduce CellPainTR, a Transformer-based architecture designed to learn foundational representations of cellular morphology.<n>Our work represents a significant step towards creating truly foundational models for image-based profiling, enabling more reliable and scalable cross-study biological analysis.
  arXiv  Detail & Related papers  (2025-09-02T03:30:07Z)
• trAIce3D: A Prompt-Driven Transformer Based U-Net for Semantic Segmentation of Microglial Cells from Large-Scale 3D Microscopy Images [39.58317527488534]
  We introduce trAIce3D, a deep-learning architecture designed for precise microglia segmentation.<n>It employs a two-stage approach: first, a 3D U-Net with vision transformers in the encoder detects somas using a sliding-window technique to cover the entire image.<n>It then refines each soma and its branches by using soma coordinates as a prompt and a 3D window around the target cell as input.<n>trained and evaluated on a dataset of 41,230 microglial cells, trAIce3D significantly improves segmentation accuracy and scalable generalization.
  arXiv  Detail & Related papers  (2025-07-30T12:54:53Z)
• CellVTA: Enhancing Vision Foundation Models for Accurate Cell Segmentation and Classification [4.307361438627596]
  We propose CellVTA (Cell Vision Transformer with Adapter), a novel method that improves the performance of vision foundation models for cell instance segmentation.<n>Our method preserves the core architecture of ViT, ensuring seamless integration with pretrained foundation models.<n>Our experiments show that CellVTA achieves 0.538 mPQ on the CoNIC dataset and 0.506 mPQ on the PanNuke dataset.
  arXiv  Detail & Related papers  (2025-04-01T13:36:46Z)
• Prompting Segment Anything Model with Domain-Adaptive Prototype for Generalizable Medical Image Segmentation [49.5901368256326]
  We propose a novel Domain-Adaptive Prompt framework for fine-tuning the Segment Anything Model (termed as DAPSAM) in segmenting medical images. Our DAPSAM achieves state-of-the-art performance on two medical image segmentation tasks with different modalities.
  arXiv  Detail & Related papers  (2024-09-19T07:28:33Z)
• Affine-Consistent Transformer for Multi-Class Cell Nuclei Detection [76.11864242047074]
  We propose a novel Affine-Consistent Transformer (AC-Former), which directly yields a sequence of nucleus positions. We introduce an Adaptive Affine Transformer (AAT) module, which can automatically learn the key spatial transformations to warp original images for local network training. Experimental results demonstrate that the proposed method significantly outperforms existing state-of-the-art algorithms on various benchmarks.
  arXiv  Detail & Related papers  (2023-10-22T02:27:02Z)
• Vision Transformers: From Semantic Segmentation to Dense Prediction [139.15562023284187]
  We explore the global context learning potentials of vision transformers (ViTs) for dense visual prediction. Our motivation is that through learning global context at full receptive field layer by layer, ViTs may capture stronger long-range dependency information. We formulate a family of Hierarchical Local-Global (HLG) Transformers, characterized by local attention within windows and global-attention across windows in a pyramidal architecture.
  arXiv  Detail & Related papers  (2022-07-19T15:49:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.