SAM-aware Test-time Adaptation for Universal Medical Image Segmentation

• URL: http://arxiv.org/abs/2506.05221v1
• Date: Thu, 05 Jun 2025 16:38:16 GMT
• Title: SAM-aware Test-time Adaptation for Universal Medical Image Segmentation
• Authors: Jianghao Wu, Yicheng Wu, Yutong Xie, Wenjia Bai, You Zhang, Feilong Tang, Yulong Li, Yasmeen George, Imran Razzak,
• Abstract summary: We propose SAM-aware Test-Time Adaptation (SAM-TTA), a pipeline that preserves the generalization of SAM while improving its segmentation performance in medical imaging via a test-time framework.<n>Our framework comprises (1) Self-adaptive Bezier Curve-based Transformation (SBCT), which adaptively converts single-channel medical images into three-channel SAM-compatible inputs while maintaining structural integrity, and (2) Dual-scale Uncertainty-driven Mean Teacher adaptation (DUMT), which employs consistency learning to align SAM's internal representations to medical semantics.
• Score: 27.96281051256966
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Universal medical image segmentation using the Segment Anything Model (SAM) remains challenging due to its limited adaptability to medical domains. Existing adaptations, such as MedSAM, enhance SAM's performance in medical imaging but at the cost of reduced generalization to unseen data. Therefore, in this paper, we propose SAM-aware Test-Time Adaptation (SAM-TTA), a fundamentally different pipeline that preserves the generalization of SAM while improving its segmentation performance in medical imaging via a test-time framework. SAM-TTA tackles two key challenges: (1) input-level discrepancies caused by differences in image acquisition between natural and medical images and (2) semantic-level discrepancies due to fundamental differences in object definition between natural and medical domains (e.g., clear boundaries vs. ambiguous structures). Specifically, our SAM-TTA framework comprises (1) Self-adaptive Bezier Curve-based Transformation (SBCT), which adaptively converts single-channel medical images into three-channel SAM-compatible inputs while maintaining structural integrity, to mitigate the input gap between medical and natural images, and (2) Dual-scale Uncertainty-driven Mean Teacher adaptation (DUMT), which employs consistency learning to align SAM's internal representations to medical semantics, enabling efficient adaptation without auxiliary supervision or expensive retraining. Extensive experiments on five public datasets demonstrate that our SAM-TTA outperforms existing TTA approaches and even surpasses fully fine-tuned models such as MedSAM in certain scenarios, establishing a new paradigm for universal medical image segmentation. Code can be found at https://github.com/JianghaoWu/SAM-TTA.

Related papers

• Few-Shot Adaptation of Training-Free Foundation Model for 3D Medical Image Segmentation [8.78725593323412]
  Few-shot Adaptation of Training-frEe SAM (FATE-SAM) is a novel method designed to adapt the advanced Segment Anything Model 2 (SAM2) for 3D medical image segmentation.<n>FATE-SAM reassembles pre-trained modules of SAM2 to enable few-shot adaptation, leveraging a small number of support examples.<n>We evaluate FATE-SAM on multiple medical imaging datasets and compare it with supervised learning methods, zero-shot SAM approaches, and fine-tuned medical SAM methods.
  arXiv  Detail & Related papers  (2025-01-15T20:44:21Z)
• Learnable Prompting SAM-induced Knowledge Distillation for Semi-supervised Medical Image Segmentation [47.789013598970925]
  We propose a learnable prompting SAM-induced Knowledge distillation framework (KnowSAM) for semi-supervised medical image segmentation.<n>Our model outperforms the state-of-the-art semi-supervised segmentation approaches.
  arXiv  Detail & Related papers  (2024-12-18T11:19:23Z)
• DB-SAM: Delving into High Quality Universal Medical Image Segmentation [100.63434169944853]
  We propose a dual-branch adapted SAM framework, named DB-SAM, to bridge the gap between natural and 2D/3D medical data. Our proposed DB-SAM achieves an absolute gain of 8.8%, compared to a recent medical SAM adapter in the literature.
  arXiv  Detail & Related papers  (2024-10-05T14:36:43Z)
• ESP-MedSAM: Efficient Self-Prompting SAM for Universal Domain-Generalized Medical Image Segmentation [18.388979166848962]
  Segment Anything Model (SAM) has demonstrated its potential in both settings. We propose an efficient self-prompting SAM for universal domain-generalized medical image segmentation, named ESP-MedSAM. ESP-MedSAM outperforms state-of-the-arts in diverse medical imaging segmentation tasks.
  arXiv  Detail & Related papers  (2024-07-19T09:32:30Z)
• Improving Segment Anything on the Fly: Auxiliary Online Learning and Adaptive Fusion for Medical Image Segmentation [52.172885882728174]
  In medical imaging contexts, it is not uncommon for human experts to rectify segmentations of specific test samples after SAM generates its segmentation predictions. We introduce a novel approach that leverages the advantages of online machine learning to enhance Segment Anything (SA) during test time. We employ rectified annotations to perform online learning, with the aim of improving the segmentation quality of SA on medical images.
  arXiv  Detail & Related papers  (2024-06-03T03:16:25Z)
• Unleashing the Potential of SAM for Medical Adaptation via Hierarchical Decoding [15.401507589312702]
  This paper introduces H-SAM, a prompt-free adaptation of the Segment Anything Model (SAM) for efficient fine-tuning of medical images. In the initial stage, H-SAM employs SAM's original decoder to generate a prior probabilistic mask, guiding a more intricate decoding process. Our H-SAM demonstrates a 4.78% improvement in average Dice compared to existing prompt-free SAM variants.
  arXiv  Detail & Related papers  (2024-03-27T05:55:16Z)
• ProMISe: Promptable Medical Image Segmentation using SAM [11.710367186709432]
  We propose an Auto-Prompting Module (APM) which provides SAM-based foundation model with Euclidean adaptive prompts in the target domain. We also propose a novel non-invasive method called Incremental Pattern Shifting (IPS) to adapt SAM to specific medical domains. By coupling these two methods, we propose ProMISe, an end-to-end non-fine-tuned framework for Promptable Medical Image.
  arXiv  Detail & Related papers  (2024-03-07T02:48:42Z)
• MA-SAM: Modality-agnostic SAM Adaptation for 3D Medical Image Segmentation [58.53672866662472]
  We introduce a modality-agnostic SAM adaptation framework, named as MA-SAM. Our method roots in the parameter-efficient fine-tuning strategy to update only a small portion of weight increments. By injecting a series of 3D adapters into the transformer blocks of the image encoder, our method enables the pre-trained 2D backbone to extract third-dimensional information from input data.
  arXiv  Detail & Related papers  (2023-09-16T02:41:53Z)
• SAM-Med2D [34.82072231983896]
  We introduce SAM-Med2D, the most comprehensive studies on applying SAM to medical 2D images. We first collect and curate approximately 4.6M images and 19.7M masks from public and private datasets. We fine-tune the encoder and decoder of the original SAM to obtain a well-performed SAM-Med2D.
  arXiv  Detail & Related papers  (2023-08-30T17:59:02Z)
• SurgicalSAM: Efficient Class Promptable Surgical Instrument Segmentation [65.52097667738884]
  We introduce SurgicalSAM, a novel end-to-end efficient-tuning approach for SAM to integrate surgical-specific information with SAM's pre-trained knowledge for improved generalisation. Specifically, we propose a lightweight prototype-based class prompt encoder for tuning, which directly generates prompt embeddings from class prototypes. In addition, to address the low inter-class variance among surgical instrument categories, we propose contrastive prototype learning.
  arXiv  Detail & Related papers  (2023-08-17T02:51:01Z)
• 3DSAM-adapter: Holistic adaptation of SAM from 2D to 3D for promptable tumor segmentation [52.699139151447945]
  We propose a novel adaptation method for transferring the segment anything model (SAM) from 2D to 3D for promptable medical image segmentation. Our model can outperform domain state-of-the-art medical image segmentation models on 3 out of 4 tasks, specifically by 8.25%, 29.87%, and 10.11% for kidney tumor, pancreas tumor, colon cancer segmentation, and achieve similar performance for liver tumor segmentation.
  arXiv  Detail & Related papers  (2023-06-23T12:09:52Z)
• Medical SAM Adapter: Adapting Segment Anything Model for Medical Image Segmentation [51.770805270588625]
  The Segment Anything Model (SAM) has recently gained popularity in the field of image segmentation. Recent studies and individual experiments have shown that SAM underperforms in medical image segmentation. We propose the Medical SAM Adapter (Med-SA), which incorporates domain-specific medical knowledge into the segmentation model.
  arXiv  Detail & Related papers  (2023-04-25T07:34:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.