Cheap Lunch for Medical Image Segmentation by Fine-tuning SAM on Few Exemplars

• URL: http://arxiv.org/abs/2308.14133v1
• Date: Sun, 27 Aug 2023 15:21:25 GMT
• Title: Cheap Lunch for Medical Image Segmentation by Fine-tuning SAM on Few Exemplars
• Authors: Weijia Feng and Lingting Zhu and Lequan Yu
• Abstract summary: The Segment Anything Model (SAM) has demonstrated remarkable capabilities of scaled-up segmentation models. However, the adoption of foundational models in the medical domain presents a challenge due to the difficulty and expense of labeling sufficient data. This paper introduces an efficient and practical approach for fine-tuning SAM using a limited number of exemplars.
• Score: 19.725817146049707
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The Segment Anything Model (SAM) has demonstrated remarkable capabilities of scaled-up segmentation models, enabling zero-shot generalization across a variety of domains. By leveraging large-scale foundational models as pre-trained models, it is a natural progression to fine-tune SAM for specific domains to further enhance performances. However, the adoption of foundational models in the medical domain presents a challenge due to the difficulty and expense of labeling sufficient data for adaptation within hospital systems. In this paper, we introduce an efficient and practical approach for fine-tuning SAM using a limited number of exemplars, making it suitable for such scenarios. Our approach combines two established techniques from the literature: an exemplar-guided synthesis module and the widely recognized Low-Rank Adaptation (LoRA) fine-tuning strategy, serving as data-level and model-level attempts respectively. Interestingly, our empirical findings suggest that SAM can be effectively aligned within the medical domain even with few labeled data. We validate our approach through experiments on brain tumor segmentation (BraTS) and multi-organ CT segmentation (Synapse). The comprehensive results underscore the feasibility and effectiveness of such an approach, paving the way for the practical application of SAM in the medical domain.

Related papers

• Improving Segment Anything on the Fly: Auxiliary Online Learning and Adaptive Fusion for Medical Image Segmentation [52.172885882728174]
  In medical imaging contexts, it is not uncommon for human experts to rectify segmentations of specific test samples after SAM generates its segmentation predictions. We introduce a novel approach that leverages the advantages of online machine learning to enhance Segment Anything (SA) during test time. We employ rectified annotations to perform online learning, with the aim of improving the segmentation quality of SA on medical images.
  arXiv  Detail & Related papers  (2024-06-03T03:16:25Z)
• How to build the best medical image segmentation algorithm using foundation models: a comprehensive empirical study with Segment Anything Model [12.051904886550956]
  This work summarizes existing fine-tuning strategies with various backbone architectures, model components, and fine-tuning algorithms across 18 combinations. We evaluate them on 17 datasets covering all common radiology modalities. We release our code and MRI-specific fine-tuned weights, which consistently obtained superior performance over the original SAM.
  arXiv  Detail & Related papers  (2024-04-15T17:31:32Z)
• Segment Any Medical Model Extended [39.80956010574076]
  We introduce SAMM Extended (SAMME), a platform that integrates new SAM variant models, adopts faster communication protocols, accommodates new interactive modes, and allows for fine-tuning of subcomponents of the models. These features can expand the potential of foundation models like SAM, and the results can be translated to applications such as image-guided therapy, mixed reality interaction, robotic navigation, and data augmentation.
  arXiv  Detail & Related papers  (2024-03-26T21:37:25Z)
• Towards a clinically accessible radiology foundation model: open-access and lightweight, with automated evaluation [113.5002649181103]
  Training open-source small multimodal models (SMMs) to bridge competency gaps for unmet clinical needs in radiology. For training, we assemble a large dataset of over 697 thousand radiology image-text pairs. For evaluation, we propose CheXprompt, a GPT-4-based metric for factuality evaluation, and demonstrate its parity with expert evaluation. The inference of LlaVA-Rad is fast and can be performed on a single V100 GPU in private settings, offering a promising state-of-the-art tool for real-world clinical applications.
  arXiv  Detail & Related papers  (2024-03-12T18:12:02Z)
• ProMISe: Promptable Medical Image Segmentation using SAM [11.710367186709432]
  We propose an Auto-Prompting Module (APM) which provides SAM-based foundation model with Euclidean adaptive prompts in the target domain. We also propose a novel non-invasive method called Incremental Pattern Shifting (IPS) to adapt SAM to specific medical domains. By coupling these two methods, we propose ProMISe, an end-to-end non-fine-tuned framework for Promptable Medical Image.
  arXiv  Detail & Related papers  (2024-03-07T02:48:42Z)
• Evaluation and improvement of Segment Anything Model for interactive histopathology image segmentation [3.677055050765245]
  The Segment Anything Model (SAM) is a foundational model for image segmentation. We evaluate SAM's performance in zero-shot and fine-tuned scenarios on histopathology data. We propose a modification of SAM's decoder to make it useful for interactive histology image segmentation.
  arXiv  Detail & Related papers  (2023-10-16T15:17:06Z)
• MA-SAM: Modality-agnostic SAM Adaptation for 3D Medical Image Segmentation [58.53672866662472]
  We introduce a modality-agnostic SAM adaptation framework, named as MA-SAM. Our method roots in the parameter-efficient fine-tuning strategy to update only a small portion of weight increments. By injecting a series of 3D adapters into the transformer blocks of the image encoder, our method enables the pre-trained 2D backbone to extract third-dimensional information from input data.
  arXiv  Detail & Related papers  (2023-09-16T02:41:53Z)
• SAM-Path: A Segment Anything Model for Semantic Segmentation in Digital Pathology [28.62539784951823]
  Foundation models, such as the Segment Anything Model (SAM), have been recently proposed for universal use in segmentation tasks. In this work, we adapt SAM for semantic segmentation by introducing trainable class prompts, followed by further enhancements through the incorporation of a pathology foundation model. Our framework, SAM-Path enhances SAM's ability to conduct semantic segmentation in digital pathology without human input prompts.
  arXiv  Detail & Related papers  (2023-07-12T20:15:25Z)
• Learnable Weight Initialization for Volumetric Medical Image Segmentation [66.3030435676252]
  We propose a learnable weight-based hybrid medical image segmentation approach. Our approach is easy to integrate into any hybrid model and requires no external training data. Experiments on multi-organ and lung cancer segmentation tasks demonstrate the effectiveness of our approach.
  arXiv  Detail & Related papers  (2023-06-15T17:55:05Z)
• Ambiguous Medical Image Segmentation using Diffusion Models [60.378180265885945]
  We introduce a single diffusion model-based approach that produces multiple plausible outputs by learning a distribution over group insights. Our proposed model generates a distribution of segmentation masks by leveraging the inherent sampling process of diffusion. Comprehensive results show that our proposed approach outperforms existing state-of-the-art ambiguous segmentation networks.
  arXiv  Detail & Related papers  (2023-04-10T17:58:22Z)
• Rethinking Semi-Supervised Medical Image Segmentation: A Variance-Reduction Perspective [51.70661197256033]
  We propose ARCO, a semi-supervised contrastive learning framework with stratified group theory for medical image segmentation. We first propose building ARCO through the concept of variance-reduced estimation and show that certain variance-reduction techniques are particularly beneficial in pixel/voxel-level segmentation tasks. We experimentally validate our approaches on eight benchmarks, i.e., five 2D/3D medical and three semantic segmentation datasets, with different label settings.
  arXiv  Detail & Related papers  (2023-02-03T13:50:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.