An Empirical Study on the Robustness of the Segment Anything Model (SAM)

• URL: http://arxiv.org/abs/2305.06422v2
• Date: Tue, 23 May 2023 20:50:07 GMT
• Title: An Empirical Study on the Robustness of the Segment Anything Model (SAM)
• Authors: Yuqing Wang, Yun Zhao, Linda Petzold
• Abstract summary: The Segment Anything Model (SAM) is a foundation model for general image segmentation. In this study we conduct a comprehensive robustness investigation of SAM under diverse real-world conditions. Our experimental results demonstrate that SAM's performance generally declines under perturbed images. By customizing prompting techniques and leveraging domain knowledge based on the unique characteristics of each dataset, the model's resilience to these perturbations can be enhanced.
• Score: 12.128991867050487
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: The Segment Anything Model (SAM) is a foundation model for general image segmentation. Although it exhibits impressive performance predominantly on natural images, understanding its robustness against various image perturbations and domains is critical for real-world applications where such challenges frequently arise. In this study we conduct a comprehensive robustness investigation of SAM under diverse real-world conditions. Our experiments encompass a wide range of image perturbations. Our experimental results demonstrate that SAM's performance generally declines under perturbed images, with varying degrees of vulnerability across different perturbations. By customizing prompting techniques and leveraging domain knowledge based on the unique characteristics of each dataset, the model's resilience to these perturbations can be enhanced, addressing dataset-specific challenges. This work sheds light on the limitations and strengths of SAM in real-world applications, promoting the development of more robust and versatile image segmentation solutions.

Related papers

• Dereflection Any Image with Diffusion Priors and Diversified Data [86.15504914121226]
  We propose a comprehensive solution with an efficient data preparation pipeline and a generalizable model for robust reflection removal. First, we introduce a dataset named Diverse Reflection Removal (DRR) created by randomly rotating reflective mediums in target scenes. Second, we propose a diffusion-based framework with one-step diffusion for deterministic outputs and fast inference.
  arXiv  Detail & Related papers  (2025-03-21T17:48:14Z)
• Segment Any-Quality Images with Generative Latent Space Enhancement [23.05638803781018]
  We propose GleSAM to boost robustness on low-quality images. We adapt the concept of latent diffusion to SAM-based segmentation frameworks. We also introduce two techniques to improve compatibility between the pre-trained diffusion model and the segmentation framework.
  arXiv  Detail & Related papers  (2025-03-16T13:58:13Z)
• UrbanSAM: Learning Invariance-Inspired Adapters for Segment Anything Models in Urban Construction [51.54946346023673]
  Urban morphology is inherently complex, with irregular objects of diverse shapes and varying scales. The Segment Anything Model (SAM) has shown significant potential in segmenting complex scenes. We propose UrbanSAM, a customized version of SAM specifically designed to analyze complex urban environments.
  arXiv  Detail & Related papers  (2025-02-21T04:25:19Z)
• Quantifying the Limits of Segmentation Foundation Models: Modeling Challenges in Segmenting Tree-Like and Low-Contrast Objects [13.311084447321234]
  This study introduces interpretable metrics quantifying object tree-likeness and textural separability. On carefully controlled synthetic experiments and real-world datasets, we show that SFM performance noticeably correlates with these factors. We link these failures to "textural confusion", where models misinterpret local structure as global texture, causing over-segmentation or difficulty distinguishing objects from similar backgrounds.
  arXiv  Detail & Related papers  (2024-12-05T15:25:51Z)
• Promptable Anomaly Segmentation with SAM Through Self-Perception Tuning [63.55145330447408]
  Segment Anything Model (SAM) has made great progress in anomaly segmentation tasks due to its impressive generalization ability. Existing methods that directly apply SAM through prompting often overlook the domain shift issue. We propose a novel Self-Perceptinon Tuning (SPT) method, aiming to enhance SAM's perception capability for anomaly segmentation.
  arXiv  Detail & Related papers  (2024-11-26T08:33:25Z)
• On Efficient Variants of Segment Anything Model: A Survey [63.127753705046]
  The Segment Anything Model (SAM) is a foundational model for image segmentation tasks, known for its strong generalization across diverse applications. To address this, a variety of SAM variants have been proposed to enhance efficiency while keeping accuracy. This survey provides the first comprehensive review of these efficient SAM variants.
  arXiv  Detail & Related papers  (2024-10-07T11:59:54Z)
• RobustSAM: Segment Anything Robustly on Degraded Images [19.767828436963317]
  Segment Anything Model (SAM) has emerged as a transformative approach in image segmentation. We propose the Robust Segment Anything Model (RobustSAM), which enhances SAM's performance on low-quality images. Our method has been shown to effectively improve the performance of SAM-based downstream tasks such as single image dehazing and deblurring.
  arXiv  Detail & Related papers  (2024-06-13T23:33:59Z)
• Towards Evaluating the Robustness of Visual State Space Models [63.14954591606638]
  Vision State Space Models (VSSMs) have demonstrated remarkable performance in visual perception tasks. However, their robustness under natural and adversarial perturbations remains a critical concern. We present a comprehensive evaluation of VSSMs' robustness under various perturbation scenarios.
  arXiv  Detail & Related papers  (2024-06-13T17:59:44Z)
• ASAM: Boosting Segment Anything Model with Adversarial Tuning [9.566046692165884]
  This paper introduces ASAM, a novel methodology that amplifies a foundation model's performance through adversarial tuning. We harness the potential of natural adversarial examples, inspired by their successful implementation in natural language processing. Our approach maintains the photorealism of adversarial examples and ensures alignment with original mask annotations.
  arXiv  Detail & Related papers  (2024-05-01T00:13:05Z)
• Rotated Multi-Scale Interaction Network for Referring Remote Sensing Image Segmentation [63.15257949821558]
  Referring Remote Sensing Image (RRSIS) is a new challenge that combines computer vision and natural language processing. Traditional Referring Image (RIS) approaches have been impeded by the complex spatial scales and orientations found in aerial imagery. We introduce the Rotated Multi-Scale Interaction Network (RMSIN), an innovative approach designed for the unique demands of RRSIS.
  arXiv  Detail & Related papers  (2023-12-19T08:14:14Z)
• Improving the Generalization of Segmentation Foundation Model under Distribution Shift via Weakly Supervised Adaptation [43.759808066264334]
  We propose a weakly supervised self-training architecture with anchor regularization and low-rank finetuning to improve the robustness and efficiency of adaptation. We validate the effectiveness on 5 types of downstream segmentation tasks including natural clean/corrupted images, medical images, camouflaged images and robotic images.
  arXiv  Detail & Related papers  (2023-12-06T13:59:22Z)
• A Survey on Segment Anything Model (SAM): Vision Foundation Model Meets Prompt Engineering [49.732628643634975]
  The Segment Anything Model (SAM), developed by Meta AI Research, offers a robust framework for image and video segmentation. This survey provides a comprehensive exploration of the SAM family, including SAM and SAM 2, highlighting their advancements in granularity and contextual understanding.
  arXiv  Detail & Related papers  (2023-05-12T07:21:59Z)
• Segment anything, from space? [8.126645790463266]
  "Segment Anything Model" (SAM) can segment objects in input imagery based on cheap input prompts. SAM usually achieved recognition accuracy similar to, or sometimes exceeding, vision models that had been trained on the target tasks. We examine whether SAM's performance extends to overhead imagery problems and help guide the community's response to its development.
  arXiv  Detail & Related papers  (2023-04-25T17:14:36Z)
• Robust Single Image Dehazing Based on Consistent and Contrast-Assisted Reconstruction [95.5735805072852]
  We propose a novel density-variational learning framework to improve the robustness of the image dehzing model. Specifically, the dehazing network is optimized under the consistency-regularized framework. Our method significantly surpasses the state-of-the-art approaches.
  arXiv  Detail & Related papers  (2022-03-29T08:11:04Z)
• A Dataset and Benchmark Towards Multi-Modal Face Anti-Spoofing Under Surveillance Scenarios [15.296568518106763]
  We propose an Attention based Face Anti-spoofing network with Feature Augment (AFA) to solve the FAS towards low-quality face images. Our model can achieve state-of-the-art performance on the CASIA-SURF dataset and our proposed GREAT-FASD-S dataset.
  arXiv  Detail & Related papers  (2021-03-29T08:14:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.