Temporally-Extended Prompts Optimization for SAM in Interactive Medical
Image Segmentation
- URL: http://arxiv.org/abs/2306.08958v1
- Date: Thu, 15 Jun 2023 08:51:24 GMT
- Title: Temporally-Extended Prompts Optimization for SAM in Interactive Medical
Image Segmentation
- Authors: Chuyun Shen, Wenhao Li, Ya Zhang, Xiangfeng Wang
- Abstract summary: The Anything Model (SAM) has emerged as a foundation model for addressing image segmentation.
This paper focuses on assessing the potential of SAM's zero-shot capabilities within the interactive medical image segmentation (IMIS) paradigm.
We develop a framework that adaptively offers suitable prompt forms for human experts.
- Score: 22.242652304004068
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: The Segmentation Anything Model (SAM) has recently emerged as a foundation
model for addressing image segmentation. Owing to the intrinsic complexity of
medical images and the high annotation cost, the medical image segmentation
(MIS) community has been encouraged to investigate SAM's zero-shot capabilities
to facilitate automatic annotation. Inspired by the extraordinary
accomplishments of interactive medical image segmentation (IMIS) paradigm, this
paper focuses on assessing the potential of SAM's zero-shot capabilities within
the IMIS paradigm to amplify its benefits in the MIS domain. Regrettably, we
observe that SAM's vulnerability to prompt forms (e.g., points, bounding boxes)
becomes notably pronounced in IMIS. This leads us to develop a framework that
adaptively offers suitable prompt forms for human experts. We refer to the
framework above as temporally-extended prompts optimization (TEPO) and model it
as a Markov decision process, solvable through reinforcement learning.
Numerical experiments on the standardized benchmark BraTS2020 demonstrate that
the learned TEPO agent can further enhance SAM's zero-shot capability in the
MIS context.
Related papers
- Improving Segment Anything on the Fly: Auxiliary Online Learning and Adaptive Fusion for Medical Image Segmentation [52.172885882728174]
In medical imaging contexts, it is not uncommon for human experts to rectify segmentations of specific test samples after SAM generates its segmentation predictions.
We introduce a novel approach that leverages the advantages of online machine learning to enhance Segment Anything (SA) during test time.
We employ rectified annotations to perform online learning, with the aim of improving the segmentation quality of SA on medical images.
arXiv Detail & Related papers (2024-06-03T03:16:25Z) - AlignSAM: Aligning Segment Anything Model to Open Context via Reinforcement Learning [61.666973416903005]
Segment Anything Model (SAM) has demonstrated its impressive generalization capabilities in open-world scenarios with the guidance of prompts.
We propose a novel framework, termed AlignSAM, designed for automatic prompting for aligning SAM to an open context.
arXiv Detail & Related papers (2024-06-01T16:21:39Z) - Segment Any Medical Model Extended [39.80956010574076]
We introduce SAMM Extended (SAMME), a platform that integrates new SAM variant models, adopts faster communication protocols, accommodates new interactive modes, and allows for fine-tuning of subcomponents of the models.
These features can expand the potential of foundation models like SAM, and the results can be translated to applications such as image-guided therapy, mixed reality interaction, robotic navigation, and data augmentation.
arXiv Detail & Related papers (2024-03-26T21:37:25Z) - ProMISe: Promptable Medical Image Segmentation using SAM [11.710367186709432]
We propose an Auto-Prompting Module (APM) which provides SAM-based foundation model with Euclidean adaptive prompts in the target domain.
We also propose a novel non-invasive method called Incremental Pattern Shifting (IPS) to adapt SAM to specific medical domains.
By coupling these two methods, we propose ProMISe, an end-to-end non-fine-tuned framework for Promptable Medical Image.
arXiv Detail & Related papers (2024-03-07T02:48:42Z) - UN-SAM: Universal Prompt-Free Segmentation for Generalized Nuclei Images [47.59627416801523]
In digital pathology, precise nuclei segmentation is pivotal yet challenged by the diversity of tissue types, staining protocols, and imaging conditions.
We propose the Universal prompt-free SAM framework for Nuclei segmentation (UN-SAM)
UN-SAM with exceptional performance surpasses state-of-the-arts in nuclei instance and semantic segmentation, especially the generalization capability in zero-shot scenarios.
arXiv Detail & Related papers (2024-02-26T15:35:18Z) - Segment Anything Model for Medical Image Segmentation: Current
Applications and Future Directions [8.216028136706948]
The recent introduction of the Segment Anything Model (SAM) signifies a noteworthy expansion of the prompt-driven paradigm into the domain of image segmentation.
We provide a comprehensive overview of recent endeavors aimed at extending the efficacy of SAM to medical image segmentation tasks.
We explore potential avenues for future research directions in SAM's role within medical image segmentation.
arXiv Detail & Related papers (2024-01-07T14:25:42Z) - Boosting Segment Anything Model Towards Open-Vocabulary Learning [69.42565443181017]
Segment Anything Model (SAM) has emerged as a new paradigmatic vision foundation model.
Despite SAM finding applications and adaptations in various domains, its primary limitation lies in the inability to grasp object semantics.
We present Sambor to seamlessly integrate SAM with the open-vocabulary object detector in an end-to-end framework.
arXiv Detail & Related papers (2023-12-06T17:19:00Z) - Self-guided Few-shot Semantic Segmentation for Remote Sensing Imagery
Based on Large Vision Models [14.292149307183967]
This research introduces a structured framework designed for the automation of few-shot semantic segmentation.
It utilizes the SAM model and facilitates a more efficient generation of semantically discernible segmentation outcomes.
Central to our methodology is a novel automatic prompt learning approach, leveraging prior guided masks to produce coarse pixel-wise prompts for SAM.
arXiv Detail & Related papers (2023-11-22T07:07:55Z) - Medical SAM Adapter: Adapting Segment Anything Model for Medical Image
Segmentation [51.770805270588625]
The Segment Anything Model (SAM) has recently gained popularity in the field of image segmentation.
Recent studies and individual experiments have shown that SAM underperforms in medical image segmentation.
We propose the Medical SAM Adapter (Med-SA), which incorporates domain-specific medical knowledge into the segmentation model.
arXiv Detail & Related papers (2023-04-25T07:34:22Z) - SAM.MD: Zero-shot medical image segmentation capabilities of the Segment
Anything Model [1.1221592576472588]
We evaluate the zero-shot capabilities of the Segment Anything Model for medical image segmentation.
We show that SAM generalizes well to CT data, making it a potential catalyst for the advancement of semi-automatic segmentation tools.
arXiv Detail & Related papers (2023-04-10T18:20:29Z) - Few-shot Medical Image Segmentation using a Global Correlation Network
with Discriminative Embedding [60.89561661441736]
We propose a novel method for few-shot medical image segmentation.
We construct our few-shot image segmentor using a deep convolutional network trained episodically.
We enhance discriminability of deep embedding to encourage clustering of the feature domains of the same class.
arXiv Detail & Related papers (2020-12-10T04:01:07Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.