AdaptiveSAM: Towards Efficient Tuning of SAM for Surgical Scene Segmentation

• URL: http://arxiv.org/abs/2308.03726v1
• Date: Mon, 7 Aug 2023 17:12:54 GMT
• Title: AdaptiveSAM: Towards Efficient Tuning of SAM for Surgical Scene Segmentation
• Authors: Jay N. Paranjape, Nithin Gopalakrishnan Nair, Shameema Sikder, S. Swaroop Vedula, Vishal M. Patel
• Abstract summary: We propose an adaptive modification of Segment-Anything (SAM) that can adjust to new datasets quickly and efficiently. AdaptiveSAM uses free-form text as prompt and can segment the object of interest with just the label name as prompt. Our experiments show that AdaptiveSAM outperforms current state-of-the-art methods on various medical imaging datasets.
• Score: 49.59991322513561
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Segmentation is a fundamental problem in surgical scene analysis using artificial intelligence. However, the inherent data scarcity in this domain makes it challenging to adapt traditional segmentation techniques for this task. To tackle this issue, current research employs pretrained models and finetunes them on the given data. Even so, these require training deep networks with millions of parameters every time new data becomes available. A recently published foundation model, Segment-Anything (SAM), generalizes well to a large variety of natural images, hence tackling this challenge to a reasonable extent. However, SAM does not generalize well to the medical domain as is without utilizing a large amount of compute resources for fine-tuning and using task-specific prompts. Moreover, these prompts are in the form of bounding-boxes or foreground/background points that need to be annotated explicitly for every image, making this solution increasingly tedious with higher data size. In this work, we propose AdaptiveSAM - an adaptive modification of SAM that can adjust to new datasets quickly and efficiently, while enabling text-prompted segmentation. For finetuning AdaptiveSAM, we propose an approach called bias-tuning that requires a significantly smaller number of trainable parameters than SAM (less than 2\%). At the same time, AdaptiveSAM requires negligible expert intervention since it uses free-form text as prompt and can segment the object of interest with just the label name as prompt. Our experiments show that AdaptiveSAM outperforms current state-of-the-art methods on various medical imaging datasets including surgery, ultrasound and X-ray. Code is available at https://github.com/JayParanjape/biastuning

Related papers

• Generalized SAM: Efficient Fine-Tuning of SAM for Variable Input Image Sizes [3.8506666685467343]
  We propose a novel efficient fine-tuning method that allows the input image size of Segment Anything Model (SAM) to be variable. Generalized SAM (GSAM) is the first to apply random cropping during training with SAM, thereby significantly reducing the computational cost of training.
  arXiv  Detail & Related papers  (2024-08-22T13:58:08Z)
• S-SAM: SVD-based Fine-Tuning of Segment Anything Model for Medical Image Segmentation [25.12190845061075]
  We propose an adaptation technique, called S-SAM, that only trains parameters equal to 0.4% of SAM's parameters and at the same time uses simply the label names as prompts for producing precise masks. We call this modified version S-SAM and evaluate it on five different modalities including endoscopic images, x-ray, ultrasound, CT, and histology images.
  arXiv  Detail & Related papers  (2024-08-12T18:53:03Z)
• Improving Segment Anything on the Fly: Auxiliary Online Learning and Adaptive Fusion for Medical Image Segmentation [52.172885882728174]
  In medical imaging contexts, it is not uncommon for human experts to rectify segmentations of specific test samples after SAM generates its segmentation predictions. We introduce a novel approach that leverages the advantages of online machine learning to enhance Segment Anything (SA) during test time. We employ rectified annotations to perform online learning, with the aim of improving the segmentation quality of SA on medical images.
  arXiv  Detail & Related papers  (2024-06-03T03:16:25Z)
• BA-SAM: Scalable Bias-Mode Attention Mask for Segment Anything Model [65.92173280096588]
  We address the challenge of image resolution variation for the Segment Anything Model (SAM) SAM, known for its zero-shot generalizability, exhibits a performance degradation when faced with datasets with varying image sizes. We present a bias-mode attention mask that allows each token to prioritize neighboring information.
  arXiv  Detail & Related papers  (2024-01-04T15:34:44Z)
• MA-SAM: Modality-agnostic SAM Adaptation for 3D Medical Image Segmentation [58.53672866662472]
  We introduce a modality-agnostic SAM adaptation framework, named as MA-SAM. Our method roots in the parameter-efficient fine-tuning strategy to update only a small portion of weight increments. By injecting a series of 3D adapters into the transformer blocks of the image encoder, our method enables the pre-trained 2D backbone to extract third-dimensional information from input data.
  arXiv  Detail & Related papers  (2023-09-16T02:41:53Z)
• Beyond Adapting SAM: Towards End-to-End Ultrasound Image Segmentation via Auto Prompting [10.308637269138146]
  We propose SAMUS as a universal model tailored for ultrasound image segmentation. We further enable it to work in an end-to-end manner denoted as AutoSAMUS. AutoSAMUS is realized by introducing an auto prompt generator (APG) to replace the manual prompt encoder of SAMUS.
  arXiv  Detail & Related papers  (2023-09-13T09:15:20Z)
• SurgicalSAM: Efficient Class Promptable Surgical Instrument Segmentation [65.52097667738884]
  We introduce SurgicalSAM, a novel end-to-end efficient-tuning approach for SAM to integrate surgical-specific information with SAM's pre-trained knowledge for improved generalisation. Specifically, we propose a lightweight prototype-based class prompt encoder for tuning, which directly generates prompt embeddings from class prototypes. In addition, to address the low inter-class variance among surgical instrument categories, we propose contrastive prototype learning.
  arXiv  Detail & Related papers  (2023-08-17T02:51:01Z)
• How to Efficiently Adapt Large Segmentation Model(SAM) to Medical Images [15.181219203629643]
  Segment Anything (SAM) exhibits impressive capabilities in zero-shot segmentation for natural images. However, when applied to medical images, SAM suffers from noticeable performance drop. In this work, we propose to freeze SAM encoder and finetune a lightweight task-specific prediction head.
  arXiv  Detail & Related papers  (2023-06-23T18:34:30Z)
• Medical SAM Adapter: Adapting Segment Anything Model for Medical Image Segmentation [51.770805270588625]
  The Segment Anything Model (SAM) has recently gained popularity in the field of image segmentation. Recent studies and individual experiments have shown that SAM underperforms in medical image segmentation. We propose the Medical SAM Adapter (Med-SA), which incorporates domain-specific medical knowledge into the segmentation model.
  arXiv  Detail & Related papers  (2023-04-25T07:34:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.