Adaptive Prompt Learning with SAM for Few-shot Scanning Probe Microscope Image Segmentation

• URL: http://arxiv.org/abs/2410.12562v1
• Date: Wed, 16 Oct 2024 13:38:01 GMT
• Title: Adaptive Prompt Learning with SAM for Few-shot Scanning Probe Microscope Image Segmentation
• Authors: Yao Shen, Ziwei Wei, Chunmeng Liu, Shuming Wei, Qi Zhao, Kaiyang Zeng, Guangyao Li,
• Abstract summary: Segment Anything Model (SAM) has demonstrated strong performance in image segmentation of natural scene images. SAM's effectiveness diminishes markedly when applied to specific scientific domains, such as Scanning Probe Microscope (SPM) images. We propose an Adaptive Prompt Learning with SAM framework tailored for few-shot SPM image segmentation.
• Score: 11.882111844381098
• License:
• Abstract: The Segment Anything Model (SAM) has demonstrated strong performance in image segmentation of natural scene images. However, its effectiveness diminishes markedly when applied to specific scientific domains, such as Scanning Probe Microscope (SPM) images. This decline in accuracy can be attributed to the distinct data distribution and limited availability of the data inherent in the scientific images. On the other hand, the acquisition of adequate SPM datasets is both time-intensive and laborious as well as skill-dependent. To address these challenges, we propose an Adaptive Prompt Learning with SAM (APL-SAM) framework tailored for few-shot SPM image segmentation. Our approach incorporates two key innovations to enhance SAM: 1) An Adaptive Prompt Learning module leverages few-shot embeddings derived from limited support set to learn adaptively central representatives, serving as visual prompts. This innovation eliminates the need for time-consuming online user interactions for providing prompts, such as exhaustively marking points and bounding boxes slice by slice; 2) A multi-source, multi-level mask decoder specifically designed for few-shot SPM image segmentation is introduced, which can effectively capture the correspondence between the support and query images. To facilitate comprehensive training and evaluation, we introduce a new dataset, SPM-Seg, curated for SPM image segmentation. Extensive experiments on this dataset reveal that the proposed APL-SAM framework significantly outperforms the original SAM, achieving over a 30% improvement in terms of Dice Similarity Coefficient with only one-shot guidance. Moreover, APL-SAM surpasses state-of-the-art few-shot segmentation methods and even fully supervised approaches in performance. Code and dataset used in this study will be made available upon acceptance.

Related papers

• CycleSAM: One-Shot Surgical Scene Segmentation using Cycle-Consistent Feature Matching to Prompt SAM [2.9500242602590565]
  CycleSAM is an approach for one-shot surgical scene segmentation using the training image-mask pair at test-time. We employ a ResNet50 encoder pretrained on surgical images in a self-supervised fashion, thereby maintaining high label-efficiency.
  arXiv  Detail & Related papers  (2024-07-09T12:08:07Z)
• SAM Fewshot Finetuning for Anatomical Segmentation in Medical Images [3.2099042811875833]
  We propose a strategy for adapting the Segment Anything (SAM) to anatomical segmentation tasks in medical images. We leverage few-shot embeddings derived from a limited set of labeled images as prompts for anatomical querying objects captured in image embeddings. Our method prioritizes the efficiency of the fine-tuning process by exclusively training the mask decoder through caching mechanisms.
  arXiv  Detail & Related papers  (2024-07-05T17:07:25Z)
• FocSAM: Delving Deeply into Focused Objects in Segmenting Anything [58.042354516491024]
  The Segment Anything Model (SAM) marks a notable milestone in segmentation models. We propose FocSAM with a pipeline redesigned on two pivotal aspects. First, we propose Dynamic Window Multi-head Self-Attention (Dwin-MSA) to dynamically refocus SAM's image embeddings on the target object. Second, we propose Pixel-wise Dynamic ReLU (P-DyReLU) to enable sufficient integration of interactive information from a few initial clicks.
  arXiv  Detail & Related papers  (2024-05-29T02:34:13Z)
• PosSAM: Panoptic Open-vocabulary Segment Anything [58.72494640363136]
  PosSAM is an open-vocabulary panoptic segmentation model that unifies the strengths of the Segment Anything Model (SAM) with the vision-native CLIP model in an end-to-end framework. We introduce a Mask-Aware Selective Ensembling (MASE) algorithm that adaptively enhances the quality of generated masks and boosts the performance of open-vocabulary classification during inference for each image.
  arXiv  Detail & Related papers  (2024-03-14T17:55:03Z)
• WSI-SAM: Multi-resolution Segment Anything Model (SAM) for histopathology whole-slide images [8.179859593451285]
  We present WSI-SAM, enhancing Segment Anything Model (SAM) with precise object segmentation capabilities for histopathology images. To fully exploit pretrained knowledge while minimizing training overhead, we keep SAM frozen, introducing only minimal extra parameters. Our model outperforms SAM by 4.1 and 2.5 percent points on a ductal carcinoma in situ (DCIS) segmentation tasks and breast cancer metastasis segmentation task.
  arXiv  Detail & Related papers  (2024-03-14T10:30:43Z)
• Rotated Multi-Scale Interaction Network for Referring Remote Sensing Image Segmentation [63.15257949821558]
  Referring Remote Sensing Image (RRSIS) is a new challenge that combines computer vision and natural language processing. Traditional Referring Image (RIS) approaches have been impeded by the complex spatial scales and orientations found in aerial imagery. We introduce the Rotated Multi-Scale Interaction Network (RMSIN), an innovative approach designed for the unique demands of RRSIS.
  arXiv  Detail & Related papers  (2023-12-19T08:14:14Z)
• Stable Segment Anything Model [79.9005670886038]
  The Segment Anything Model (SAM) achieves remarkable promptable segmentation given high-quality prompts. This paper presents the first comprehensive analysis on SAM's segmentation stability across a diverse spectrum of prompt qualities. Our solution, termed Stable-SAM, offers several advantages: 1) improved SAM's segmentation stability across a wide range of prompt qualities, while 2) retaining SAM's powerful promptable segmentation efficiency and generality.
  arXiv  Detail & Related papers  (2023-11-27T12:51:42Z)
• Self-guided Few-shot Semantic Segmentation for Remote Sensing Imagery Based on Large Vision Models [14.292149307183967]
  This research introduces a structured framework designed for the automation of few-shot semantic segmentation. It utilizes the SAM model and facilitates a more efficient generation of semantically discernible segmentation outcomes. Central to our methodology is a novel automatic prompt learning approach, leveraging prior guided masks to produce coarse pixel-wise prompts for SAM.
  arXiv  Detail & Related papers  (2023-11-22T07:07:55Z)
• RefSAM: Efficiently Adapting Segmenting Anything Model for Referring Video Object Segmentation [53.4319652364256]
  This paper presents the RefSAM model, which explores the potential of SAM for referring video object segmentation. Our proposed approach adapts the original SAM model to enhance cross-modality learning by employing a lightweight Cross-RValModal. We employ a parameter-efficient tuning strategy to align and fuse the language and vision features effectively.
  arXiv  Detail & Related papers  (2023-07-03T13:21:58Z)
• RSPrompter: Learning to Prompt for Remote Sensing Instance Segmentation based on Visual Foundation Model [29.42043345787285]
  We propose a method to learn the generation of appropriate prompts for Segment Anything Model (SAM) This enables SAM to produce semantically discernible segmentation results for remote sensing images. We also propose several ongoing derivatives for instance segmentation tasks, drawing on recent advancements within the SAM community, and compare their performance with RSPrompter.
  arXiv  Detail & Related papers  (2023-06-28T14:51:34Z)
• Learning Self-Supervised Low-Rank Network for Single-Stage Weakly and Semi-Supervised Semantic Segmentation [119.009033745244]
  This paper presents a Self-supervised Low-Rank Network ( SLRNet) for single-stage weakly supervised semantic segmentation (WSSS) and semi-supervised semantic segmentation (SSSS) SLRNet uses cross-view self-supervision, that is, it simultaneously predicts several attentive LR representations from different views of an image to learn precise pseudo-labels. Experiments on the Pascal VOC 2012, COCO, and L2ID datasets demonstrate that our SLRNet outperforms both state-of-the-art WSSS and SSSS methods with a variety of different settings.
  arXiv  Detail & Related papers  (2022-03-19T09:19:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.