SAM Fails to Segment Anything? -- SAM-Adapter: Adapting SAM in
Underperformed Scenes: Camouflage, Shadow, Medical Image Segmentation, and
More
- URL: http://arxiv.org/abs/2304.09148v3
- Date: Tue, 2 May 2023 17:06:51 GMT
- Title: SAM Fails to Segment Anything? -- SAM-Adapter: Adapting SAM in
Underperformed Scenes: Camouflage, Shadow, Medical Image Segmentation, and
More
- Authors: Tianrun Chen, Lanyun Zhu, Chaotao Ding, Runlong Cao, Yan Wang, Zejian
Li, Lingyun Sun, Papa Mao, Ying Zang
- Abstract summary: We propose textbfSAM-Adapter, which incorporates domain-specific information or visual prompts into the segmentation network by using simple yet effective adapters.
We can even outperform task-specific network models and achieve state-of-the-art performance in the task we tested: camouflaged object detection.
- Score: 13.047310918166762
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: The emergence of large models, also known as foundation models, has brought
significant advancements to AI research. One such model is Segment Anything
(SAM), which is designed for image segmentation tasks. However, as with other
foundation models, our experimental findings suggest that SAM may fail or
perform poorly in certain segmentation tasks, such as shadow detection and
camouflaged object detection (concealed object detection). This study first
paves the way for applying the large pre-trained image segmentation model SAM
to these downstream tasks, even in situations where SAM performs poorly. Rather
than fine-tuning the SAM network, we propose \textbf{SAM-Adapter}, which
incorporates domain-specific information or visual prompts into the
segmentation network by using simple yet effective adapters. By integrating
task-specific knowledge with general knowledge learnt by the large model,
SAM-Adapter can significantly elevate the performance of SAM in challenging
tasks as shown in extensive experiments. We can even outperform task-specific
network models and achieve state-of-the-art performance in the task we tested:
camouflaged object detection, shadow detection. We also tested polyp
segmentation (medical image segmentation) and achieves better results. We
believe our work opens up opportunities for utilizing SAM in downstream tasks,
with potential applications in various fields, including medical image
processing, agriculture, remote sensing, and more.
Related papers
- MAS-SAM: Segment Any Marine Animal with Aggregated Features [55.91291540810978]
We propose a novel feature learning framework named MAS-SAM for marine animal segmentation.
Our method enables to extract richer marine information from global contextual cues to fine-grained local details.
arXiv Detail & Related papers (2024-04-24T07:38:14Z) - Performance Evaluation of Segment Anything Model with Variational Prompting for Application to Non-Visible Spectrum Imagery [15.748043194987075]
This work assesses Segment Anything Model capabilities in segmenting objects of interest in the X-ray/infrared modalities.
Our results show that SAM can segment objects in the X-ray modality when given a box prompt, but its performance varies for point prompts.
We find that infrared objects are also challenging to segment with point prompts given the low-contrast nature of this modality.
arXiv Detail & Related papers (2024-04-18T16:04:14Z) - Deep Instruction Tuning for Segment Anything Model [68.7934961590075]
Segment Anything Model (SAM) has become a research hotspot in the fields of multimedia and computer vision.
SAM can support different types of segmentation prompts, but it performs much worse on text-instructed tasks.
We propose two simple yet effective deep instruction tuning (DIT) methods for SAM, one is end-to-end and the other is layer-wise.
arXiv Detail & Related papers (2024-03-31T11:37:43Z) - RSAM-Seg: A SAM-based Approach with Prior Knowledge Integration for
Remote Sensing Image Semantic Segmentation [10.37240769959699]
Segment Anything Model (SAM) provides a universal pre-training model for image segmentation tasks.
We propose RSAM-Seg, which stands for Remote Sensing SAM with Semantic, as a tailored modification of SAM for the remote sensing field.
Adapter-Scale, a set of supplementary scaling modules, are proposed in the multi-head attention blocks of the encoder part of SAM.
Experiments are conducted on four distinct remote sensing scenarios, encompassing cloud detection, field monitoring, building detection and road mapping tasks.
arXiv Detail & Related papers (2024-02-29T09:55:46Z) - TinySAM: Pushing the Envelope for Efficient Segment Anything Model [76.21007576954035]
We propose a framework to obtain a tiny segment anything model (TinySAM) while maintaining the strong zero-shot performance.
We first propose a full-stage knowledge distillation method with hard prompt sampling and hard mask weighting strategy to distill a lightweight student model.
We also adapt the post-training quantization to the promptable segmentation task and further reduce the computational cost.
arXiv Detail & Related papers (2023-12-21T12:26:11Z) - When SAM Meets Sonar Images [6.902760999492406]
Segment Anything Model (SAM) has revolutionized the way of segmentation.
SAM's performance may decline when applied to tasks involving domains that differ from natural images.
By employing fine-tuning techniques, SAM exhibits promising capabilities in specific domains, such as medicine and planetary science.
arXiv Detail & Related papers (2023-06-25T03:15:14Z) - AutoSAM: Adapting SAM to Medical Images by Overloading the Prompt
Encoder [101.28268762305916]
In this work, we replace Segment Anything Model with an encoder that operates on the same input image.
We obtain state-of-the-art results on multiple medical images and video benchmarks.
For inspecting the knowledge within it, and providing a lightweight segmentation solution, we also learn to decode it into a mask by a shallow deconvolution network.
arXiv Detail & Related papers (2023-06-10T07:27:00Z) - Personalize Segment Anything Model with One Shot [52.54453744941516]
We propose a training-free Personalization approach for Segment Anything Model (SAM)
Given only a single image with a reference mask, PerSAM first localizes the target concept by a location prior.
PerSAM segments it within other images or videos via three techniques: target-guided attention, target-semantic prompting, and cascaded post-refinement.
arXiv Detail & Related papers (2023-05-04T17:59:36Z) - Segment anything, from space? [8.126645790463266]
"Segment Anything Model" (SAM) can segment objects in input imagery based on cheap input prompts.
SAM usually achieved recognition accuracy similar to, or sometimes exceeding, vision models that had been trained on the target tasks.
We examine whether SAM's performance extends to overhead imagery problems and help guide the community's response to its development.
arXiv Detail & Related papers (2023-04-25T17:14:36Z) - Medical SAM Adapter: Adapting Segment Anything Model for Medical Image
Segmentation [51.770805270588625]
The Segment Anything Model (SAM) has recently gained popularity in the field of image segmentation.
Recent studies and individual experiments have shown that SAM underperforms in medical image segmentation.
We propose the Medical SAM Adapter (Med-SA), which incorporates domain-specific medical knowledge into the segmentation model.
arXiv Detail & Related papers (2023-04-25T07:34:22Z) - Can SAM Segment Anything? When SAM Meets Camouflaged Object Detection [8.476593072868056]
SAM is a segmentation model recently released by Meta AI Research.
We try to ask if SAM can address the camouflage object detection (COD) task and evaluate the performance of SAM on the COD benchmark.
We also compare SAM's performance with 22 state-of-the-art COD methods.
arXiv Detail & Related papers (2023-04-10T17:05:58Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.