All-in-SAM: from Weak Annotation to Pixel-wise Nuclei Segmentation with Prompt-based Finetuning

• URL: http://arxiv.org/abs/2307.00290v2
• Date: Tue, 29 Aug 2023 03:31:58 GMT
• Title: All-in-SAM: from Weak Annotation to Pixel-wise Nuclei Segmentation with Prompt-based Finetuning
• Authors: Can Cui, Ruining Deng, Quan Liu, Tianyuan Yao, Shunxing Bao, Lucas W. Remedios, Yucheng Tang, Yuankai Huo
• Abstract summary: The Segment Anything Model (SAM) is a recently proposed prompt-based segmentation model in a generic zero-shot segmentation approach. We introduce a pipeline that utilizes the SAM through the entire AI development workflow without requiring manual prompts during the inference stage. Our experimental results reveal two key findings: 1) the proposed pipeline surpasses the state-of-the-art (SOTA) methods in a nuclei segmentation task on the public Monuseg dataset, and 2) the utilization of weak and few annotations for SAM finetuning achieves competitive performance.
• Score: 16.016139980843835
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The Segment Anything Model (SAM) is a recently proposed prompt-based segmentation model in a generic zero-shot segmentation approach. With the zero-shot segmentation capacity, SAM achieved impressive flexibility and precision on various segmentation tasks. However, the current pipeline requires manual prompts during the inference stage, which is still resource intensive for biomedical image segmentation. In this paper, instead of using prompts during the inference stage, we introduce a pipeline that utilizes the SAM, called all-in-SAM, through the entire AI development workflow (from annotation generation to model finetuning) without requiring manual prompts during the inference stage. Specifically, SAM is first employed to generate pixel-level annotations from weak prompts (e.g., points, bounding box). Then, the pixel-level annotations are used to finetune the SAM segmentation model rather than training from scratch. Our experimental results reveal two key findings: 1) the proposed pipeline surpasses the state-of-the-art (SOTA) methods in a nuclei segmentation task on the public Monuseg dataset, and 2) the utilization of weak and few annotations for SAM finetuning achieves competitive performance compared to using strong pixel-wise annotated data.

Related papers

• SAM-SP: Self-Prompting Makes SAM Great Again [11.109389094334894]
  Segment Anything Model (SAM) has demonstrated impressive capabilities in zero-shot segmentation tasks. SAM encounters noticeably degradation performance when applied to specific domains, such as medical images. We introduce a novel self-prompting based fine-tuning approach, called SAM-SP, tailored for extending the vanilla SAM model.
  arXiv  Detail & Related papers  (2024-08-22T13:03:05Z)
• AlignSAM: Aligning Segment Anything Model to Open Context via Reinforcement Learning [61.666973416903005]
  Segment Anything Model (SAM) has demonstrated its impressive generalization capabilities in open-world scenarios with the guidance of prompts. We propose a novel framework, termed AlignSAM, designed for automatic prompting for aligning SAM to an open context.
  arXiv  Detail & Related papers  (2024-06-01T16:21:39Z)
• Moving Object Segmentation: All You Need Is SAM (and Flow) [82.78026782967959]
  We investigate two models for combining SAM with optical flow that harness the segmentation power of SAM with the ability of flow to discover and group moving objects. In the first model, we adapt SAM to take optical flow, rather than RGB, as an input. In the second, SAM takes RGB as an input, and flow is used as a segmentation prompt. These surprisingly simple methods, without any further modifications, outperform all previous approaches by a considerable margin in both single and multi-object benchmarks.
  arXiv  Detail & Related papers  (2024-04-18T17:59:53Z)
• PosSAM: Panoptic Open-vocabulary Segment Anything [58.72494640363136]
  PosSAM is an open-vocabulary panoptic segmentation model that unifies the strengths of the Segment Anything Model (SAM) with the vision-native CLIP model in an end-to-end framework. We introduce a Mask-Aware Selective Ensembling (MASE) algorithm that adaptively enhances the quality of generated masks and boosts the performance of open-vocabulary classification during inference for each image.
  arXiv  Detail & Related papers  (2024-03-14T17:55:03Z)
• BLO-SAM: Bi-level Optimization Based Overfitting-Preventing Finetuning of SAM [37.1263294647351]
  We introduce BLO-SAM, which finetunes the Segment Anything Model (SAM) based on bi-level optimization (BLO) BLO-SAM reduces the risk of overfitting by training the model's weight parameters and the prompt embedding on two separate subsets of the training dataset. Results demonstrate BLO-SAM's superior performance over various state-of-the-art image semantic segmentation methods.
  arXiv  Detail & Related papers  (2024-02-26T06:36:32Z)
• TinySAM: Pushing the Envelope for Efficient Segment Anything Model [73.06322749886483]
  We propose a framework to obtain a tiny segment anything model (TinySAM) while maintaining the strong zero-shot performance. With all these proposed methods, our TinySAM leads to orders of magnitude computational reduction and pushes the envelope for efficient segment anything task.
  arXiv  Detail & Related papers  (2023-12-21T12:26:11Z)
• Guided Prompting in SAM for Weakly Supervised Cell Segmentation in Histopathological Images [27.14641973632063]
  This paper focuses on using weak supervision -- annotation from related tasks -- to induce a segmenter. Recent foundation models, such as Segment Anything (SAM), can use prompts to leverage additional supervision during inference. All SAM-based solutions hugely outperform existing weakly supervised image segmentation models, obtaining 9-15 pt Dice gains.
  arXiv  Detail & Related papers  (2023-11-29T11:18:48Z)
• Stable Segment Anything Model [79.9005670886038]
  The Segment Anything Model (SAM) achieves remarkable promptable segmentation given high-quality prompts. This paper presents the first comprehensive analysis on SAM's segmentation stability across a diverse spectrum of prompt qualities. Our solution, termed Stable-SAM, offers several advantages: 1) improved SAM's segmentation stability across a wide range of prompt qualities, while 2) retaining SAM's powerful promptable segmentation efficiency and generality.
  arXiv  Detail & Related papers  (2023-11-27T12:51:42Z)
• SurgicalSAM: Efficient Class Promptable Surgical Instrument Segmentation [65.52097667738884]
  We introduce SurgicalSAM, a novel end-to-end efficient-tuning approach for SAM to integrate surgical-specific information with SAM's pre-trained knowledge for improved generalisation. Specifically, we propose a lightweight prototype-based class prompt encoder for tuning, which directly generates prompt embeddings from class prototypes. In addition, to address the low inter-class variance among surgical instrument categories, we propose contrastive prototype learning.
  arXiv  Detail & Related papers  (2023-08-17T02:51:01Z)
• RSPrompter: Learning to Prompt for Remote Sensing Instance Segmentation based on Visual Foundation Model [29.42043345787285]
  We propose a method to learn the generation of appropriate prompts for Segment Anything Model (SAM) This enables SAM to produce semantically discernible segmentation results for remote sensing images. We also propose several ongoing derivatives for instance segmentation tasks, drawing on recent advancements within the SAM community, and compare their performance with RSPrompter.
  arXiv  Detail & Related papers  (2023-06-28T14:51:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.