From Specialist to Generalist: Unlocking SAM's Learning Potential on Unlabeled Medical Images

• URL: http://arxiv.org/abs/2601.17934v2
• Date: Wed, 28 Jan 2026 18:55:46 GMT
• Title: From Specialist to Generalist: Unlocking SAM's Learning Potential on Unlabeled Medical Images
• Authors: Vi Vu, Thanh-Huy Nguyen, Tien-Thinh Nguyen, Ba-Thinh Lam, Hoang-Thien Nguyen, Tianyang Wang, Xingjian Li, Min Xu,
• Abstract summary: We introduce SC-SAM, a specialist-generalist framework where U-Net provides point-based prompts and pseudo-labels to guide SAM's adaptation.<n>This reciprocal guidance forms a bidirectional co-training loop that allows both models to effectively exploit the unlabeled data.<n>Our method achieves state-of-the-art results, outperforming other existing semi-supervised SAM variants and even medical foundation models like MedSAM.
• Score: 12.062960289184199
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Foundation models like the Segment Anything Model (SAM) show strong generalization, yet adapting them to medical images remains difficult due to domain shift, scarce labels, and the inability of Parameter-Efficient Fine-Tuning (PEFT) to exploit unlabeled data. While conventional models like U-Net excel in semi-supervised medical learning, their potential to assist a PEFT SAM has been largely overlooked. We introduce SC-SAM, a specialist-generalist framework where U-Net provides point-based prompts and pseudo-labels to guide SAM's adaptation, while SAM serves as a powerful generalist supervisor to regularize U-Net. This reciprocal guidance forms a bidirectional co-training loop that allows both models to effectively exploit the unlabeled data. Across prostate MRI and polyp segmentation benchmarks, our method achieves state-of-the-art results, outperforming other existing semi-supervised SAM variants and even medical foundation models like MedSAM, highlighting the value of specialist-generalist cooperation for label-efficient medical image segmentation. Our code is available at https://github.com/vnlvi2k3/SC-SAM.

Related papers

• SAMora: Enhancing SAM through Hierarchical Self-Supervised Pre-Training for Medical Images [19.136303136685207]
  We propose SAMora, a framework that captures hierarchical medical knowledge at the image, patch, and pixel levels.<n> SAMora is compatible with various SAM variants, including SAM2, SAMed, and H-SAM.<n>It achieves state-of-the-art performance in both few-shot and fully supervised settings while reducing fine-tuning epochs by 90%.
  arXiv  Detail & Related papers  (2025-11-09T03:54:51Z)
• VesSAM: Efficient Multi-Prompting for Segmenting Complex Vessel [68.24765319399286]
  We present VesSAM, a powerful and efficient framework tailored for 2D vessel segmentation.<n>VesSAM integrates (1) a convolutional adapter to enhance local texture features, (2) a multi-prompt encoder that fuses anatomical prompts, and (3) a lightweight mask decoder to reduce jagged artifacts.<n>VesSAM consistently outperforms state-of-the-art PEFT-based SAM variants by over 10% Dice and 13% IoU.
  arXiv  Detail & Related papers  (2025-11-02T15:47:05Z)
• SAM-aware Test-time Adaptation for Universal Medical Image Segmentation [27.96281051256966]
  We propose SAM-aware Test-Time Adaptation (SAM-TTA), a pipeline that preserves the generalization of SAM while improving its segmentation performance in medical imaging via a test-time framework.<n>Our framework comprises (1) Self-adaptive Bezier Curve-based Transformation (SBCT), which adaptively converts single-channel medical images into three-channel SAM-compatible inputs while maintaining structural integrity, and (2) Dual-scale Uncertainty-driven Mean Teacher adaptation (DUMT), which employs consistency learning to align SAM's internal representations to medical semantics.
  arXiv  Detail & Related papers  (2025-06-05T16:38:16Z)
• WeakMedSAM: Weakly-Supervised Medical Image Segmentation via SAM with Sub-Class Exploration and Prompt Affinity Mining [31.81408955413914]
  We investigate a weakly-supervised SAM-based segmentation model, namely WeakMedSAM, to reduce the labeling cost.<n>Specifically, our proposed WeakMedSAM contains two modules: 1) to mitigate severe co-occurrence in medical images, and 2) to improve the quality of the class activation maps.<n>Our method can be applied to any SAM-like backbone, and we conduct experiments with SAMUS and EfficientSAM.
  arXiv  Detail & Related papers  (2025-03-06T05:28:44Z)
• Learnable Prompting SAM-induced Knowledge Distillation for Semi-supervised Medical Image Segmentation [47.789013598970925]
  We propose a learnable prompting SAM-induced Knowledge distillation framework (KnowSAM) for semi-supervised medical image segmentation.<n>Our model outperforms the state-of-the-art semi-supervised segmentation approaches.
  arXiv  Detail & Related papers  (2024-12-18T11:19:23Z)
• De-LightSAM: Modality-Decoupled Lightweight SAM for Generalizable Medical Segmentation [28.16884929151585]
  We propose a modality-decoupled lightweight SAM for domain-generalized medical image segmentation.<n> Specifically, we first devise a lightweight domain-controllable image encoder (DC-Encoder) that produces discriminative visual features for diverse modalities.<n>Finally, we design the query-decoupled modality decoder (QM-Decoder) that leverages a one-to-one strategy to provide an independent decoding channel.
  arXiv  Detail & Related papers  (2024-07-19T09:32:30Z)
• Segment Anything without Supervision [65.93211374889196]
  We present Unsupervised SAM (UnSAM) for promptable and automatic whole-image segmentation. UnSAM utilizes a divide-and-conquer strategy to "discover" the hierarchical structure of visual scenes. We show that supervised SAM can also benefit from our self-supervised labels.
  arXiv  Detail & Related papers  (2024-06-28T17:47:32Z)
• Unleashing the Potential of SAM for Medical Adaptation via Hierarchical Decoding [15.401507589312702]
  This paper introduces H-SAM, a prompt-free adaptation of the Segment Anything Model (SAM) for efficient fine-tuning of medical images. In the initial stage, H-SAM employs SAM's original decoder to generate a prior probabilistic mask, guiding a more intricate decoding process. Our H-SAM demonstrates a 4.78% improvement in average Dice compared to existing prompt-free SAM variants.
  arXiv  Detail & Related papers  (2024-03-27T05:55:16Z)
• BoxCell: Leveraging SAM for Cell Segmentation with Box Supervision [22.761067372749398]
  We study a weakly supervised setting, where only bounding box supervision is available, and present the use of Segment Anything (SAM) for this without any finetuning.<n>We propose BoxCell, a cell segmentation framework that utilizes SAM's capability to interpret bounding boxes as prompts, emphboth at train and test times.<n>We experiment on three publicly available cell segmentation datasets namely, CoNSep, MoNuSeg, and TNBC, and find that BoxCell significantly outperforms existing box supervised image segmentation models.
  arXiv  Detail & Related papers  (2023-11-29T11:18:48Z)
• MA-SAM: Modality-agnostic SAM Adaptation for 3D Medical Image Segmentation [58.53672866662472]
  We introduce a modality-agnostic SAM adaptation framework, named as MA-SAM. Our method roots in the parameter-efficient fine-tuning strategy to update only a small portion of weight increments. By injecting a series of 3D adapters into the transformer blocks of the image encoder, our method enables the pre-trained 2D backbone to extract third-dimensional information from input data.
  arXiv  Detail & Related papers  (2023-09-16T02:41:53Z)
• SurgicalSAM: Efficient Class Promptable Surgical Instrument Segmentation [65.52097667738884]
  We introduce SurgicalSAM, a novel end-to-end efficient-tuning approach for SAM to integrate surgical-specific information with SAM's pre-trained knowledge for improved generalisation. Specifically, we propose a lightweight prototype-based class prompt encoder for tuning, which directly generates prompt embeddings from class prototypes. In addition, to address the low inter-class variance among surgical instrument categories, we propose contrastive prototype learning.
  arXiv  Detail & Related papers  (2023-08-17T02:51:01Z)
• Medical SAM Adapter: Adapting Segment Anything Model for Medical Image Segmentation [51.770805270588625]
  The Segment Anything Model (SAM) has recently gained popularity in the field of image segmentation. Recent studies and individual experiments have shown that SAM underperforms in medical image segmentation. We propose the Medical SAM Adapter (Med-SA), which incorporates domain-specific medical knowledge into the segmentation model.
  arXiv  Detail & Related papers  (2023-04-25T07:34:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.