An Empirical Study on the Robustness of the Segment Anything Model (SAM)
- URL: http://arxiv.org/abs/2305.06422v2
- Date: Tue, 23 May 2023 20:50:07 GMT
- Title: An Empirical Study on the Robustness of the Segment Anything Model (SAM)
- Authors: Yuqing Wang, Yun Zhao, Linda Petzold
- Abstract summary: The Segment Anything Model (SAM) is a foundation model for general image segmentation.
In this study we conduct a comprehensive robustness investigation of SAM under diverse real-world conditions.
Our experimental results demonstrate that SAM's performance generally declines under perturbed images.
By customizing prompting techniques and leveraging domain knowledge based on the unique characteristics of each dataset, the model's resilience to these perturbations can be enhanced.
- Score: 12.128991867050487
- License: http://creativecommons.org/licenses/by-nc-sa/4.0/
- Abstract: The Segment Anything Model (SAM) is a foundation model for general image
segmentation. Although it exhibits impressive performance predominantly on
natural images, understanding its robustness against various image
perturbations and domains is critical for real-world applications where such
challenges frequently arise. In this study we conduct a comprehensive
robustness investigation of SAM under diverse real-world conditions. Our
experiments encompass a wide range of image perturbations. Our experimental
results demonstrate that SAM's performance generally declines under perturbed
images, with varying degrees of vulnerability across different perturbations.
By customizing prompting techniques and leveraging domain knowledge based on
the unique characteristics of each dataset, the model's resilience to these
perturbations can be enhanced, addressing dataset-specific challenges. This
work sheds light on the limitations and strengths of SAM in real-world
applications, promoting the development of more robust and versatile image
segmentation solutions.
Related papers
- RobustSAM: Segment Anything Robustly on Degraded Images [19.767828436963317]
Segment Anything Model (SAM) has emerged as a transformative approach in image segmentation.
We propose the Robust Segment Anything Model (RobustSAM), which enhances SAM's performance on low-quality images.
Our method has been shown to effectively improve the performance of SAM-based downstream tasks such as single image dehazing and deblurring.
arXiv Detail & Related papers (2024-06-13T23:33:59Z) - Towards Evaluating the Robustness of Visual State Space Models [63.14954591606638]
Vision State Space Models (VSSMs) have demonstrated remarkable performance in visual perception tasks.
However, their robustness under natural and adversarial perturbations remains a critical concern.
We present a comprehensive evaluation of VSSMs' robustness under various perturbation scenarios.
arXiv Detail & Related papers (2024-06-13T17:59:44Z) - ASAM: Boosting Segment Anything Model with Adversarial Tuning [9.566046692165884]
This paper introduces ASAM, a novel methodology that amplifies a foundation model's performance through adversarial tuning.
We harness the potential of natural adversarial examples, inspired by their successful implementation in natural language processing.
Our approach maintains the photorealism of adversarial examples and ensures alignment with original mask annotations.
arXiv Detail & Related papers (2024-05-01T00:13:05Z) - A SAM-guided Two-stream Lightweight Model for Anomaly Detection [50.28310943263051]
We propose a SAM-guided Two-stream Lightweight Model for unsupervised anomaly detection (STLM)
Our experiments conducted on MVTec AD benchmark show that STLM, with about 16M parameters and achieving an inference time in 20ms, competes effectively with state-of-the-art methods.
arXiv Detail & Related papers (2024-02-29T13:29:10Z) - Rotated Multi-Scale Interaction Network for Referring Remote Sensing Image Segmentation [63.15257949821558]
Referring Remote Sensing Image (RRSIS) is a new challenge that combines computer vision and natural language processing.
Traditional Referring Image (RIS) approaches have been impeded by the complex spatial scales and orientations found in aerial imagery.
We introduce the Rotated Multi-Scale Interaction Network (RMSIN), an innovative approach designed for the unique demands of RRSIS.
arXiv Detail & Related papers (2023-12-19T08:14:14Z) - Improving the Generalization of Segmentation Foundation Model under Distribution Shift via Weakly Supervised Adaptation [43.759808066264334]
We propose a weakly supervised self-training architecture with anchor regularization and low-rank finetuning to improve the robustness and efficiency of adaptation.
We validate the effectiveness on 5 types of downstream segmentation tasks including natural clean/corrupted images, medical images, camouflaged images and robotic images.
arXiv Detail & Related papers (2023-12-06T13:59:22Z) - Steerable Conditional Diffusion for Out-of-Distribution Adaptation in
Imaging Inverse Problems [78.76955228709241]
We introduce a novel sampling framework called Steerable Conditional Diffusion.
This framework adapts the denoising network specifically to the available measured data.
We achieve substantial enhancements in OOD performance across diverse imaging modalities.
arXiv Detail & Related papers (2023-08-28T08:47:06Z) - Segment anything, from space? [8.126645790463266]
"Segment Anything Model" (SAM) can segment objects in input imagery based on cheap input prompts.
SAM usually achieved recognition accuracy similar to, or sometimes exceeding, vision models that had been trained on the target tasks.
We examine whether SAM's performance extends to overhead imagery problems and help guide the community's response to its development.
arXiv Detail & Related papers (2023-04-25T17:14:36Z) - Robust Single Image Dehazing Based on Consistent and Contrast-Assisted
Reconstruction [95.5735805072852]
We propose a novel density-variational learning framework to improve the robustness of the image dehzing model.
Specifically, the dehazing network is optimized under the consistency-regularized framework.
Our method significantly surpasses the state-of-the-art approaches.
arXiv Detail & Related papers (2022-03-29T08:11:04Z) - Salient Objects in Clutter [130.63976772770368]
This paper identifies and addresses a serious design bias of existing salient object detection (SOD) datasets.
This design bias has led to a saturation in performance for state-of-the-art SOD models when evaluated on existing datasets.
We propose a new high-quality dataset and update the previous saliency benchmark.
arXiv Detail & Related papers (2021-05-07T03:49:26Z) - A Dataset and Benchmark Towards Multi-Modal Face Anti-Spoofing Under
Surveillance Scenarios [15.296568518106763]
We propose an Attention based Face Anti-spoofing network with Feature Augment (AFA) to solve the FAS towards low-quality face images.
Our model can achieve state-of-the-art performance on the CASIA-SURF dataset and our proposed GREAT-FASD-S dataset.
arXiv Detail & Related papers (2021-03-29T08:14:14Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.