ProSAM: Enhancing the Robustness of SAM-based Visual Reference Segmentation with Probabilistic Prompts

• URL: http://arxiv.org/abs/2506.21835v3
• Date: Sun, 03 Aug 2025 08:12:39 GMT
• Title: ProSAM: Enhancing the Robustness of SAM-based Visual Reference Segmentation with Probabilistic Prompts
• Authors: Xiaoqi Wang, Clint Sebastian, Wenbin He, Liu Ren,
• Abstract summary: We introduce ProSAM, a simple but effective method to address the stability challenges we identified in existing SAM-based visual reference segmentation approaches.<n>ProSAM avoids generating prompts that lie in unstable regions, overcoming the instability caused by less robust prompts.<n>Our approach consistently surpasses state-of-the-art methods on the Pascal-5$i$ and COCO-20$i$ datasets.
• Score: 15.582637232358177
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The recent advancements in large foundation models have driven the success of open-set image segmentation, a task focused on segmenting objects beyond predefined categories. Among various prompt types (such as points, boxes, texts, and visual references), visual reference segmentation stands out for its unique flexibility and strong zero-shot capabilities. Recently, several SAM-based methods have made notable progress in this task by automatically generating prompts to guide SAM. However, these methods often generate prompts at boundaries of target regions due to suboptimal prompt encoder, which results in instability and reduced robustness. In this work, we introduce ProSAM, a simple but effective method to address the stability challenges we identified in existing SAM-based visual reference segmentation approaches. By learning a variational prompt encoder to predict multivariate prompt distributions, ProSAM avoids generating prompts that lie in unstable regions, overcoming the instability caused by less robust prompts. Our approach consistently surpasses state-of-the-art methods on the Pascal-5$^i$ and COCO-20$^i$ datasets, providing a more robust solution for visual reference segmentation.

Related papers

• ConformalSAM: Unlocking the Potential of Foundational Segmentation Models in Semi-Supervised Semantic Segmentation with Conformal Prediction [57.930531826380836]
  This work explores whether a foundational segmentation model can address label scarcity in the pixel-level vision task as an annotator for unlabeled images.<n>We propose ConformalSAM, a novel SSSS framework which first calibrates the foundation model using the target domain's labeled data and then filters out unreliable pixel labels of unlabeled data.
  arXiv  Detail & Related papers  (2025-07-21T17:02:57Z)
• Segment Concealed Objects with Incomplete Supervision [63.637733655439334]
  Incompletely-Supervised Concealed Object (ISCOS) involves segmenting objects that seamlessly blend into their surrounding environments.<n>This task remains highly challenging due to the limited supervision provided by the incompletely annotated training data.<n>In this paper, we introduce the first unified method for ISCOS to address these challenges.
  arXiv  Detail & Related papers  (2025-06-10T16:25:15Z)
• S^4M: Boosting Semi-Supervised Instance Segmentation with SAM [25.94737539065708]
  Semi-supervised instance segmentation poses challenges due to limited labeled data.<n>Current teacher-student frameworks still suffer from performance constraints due to unreliable pseudo-label quality.
  arXiv  Detail & Related papers  (2025-04-07T17:59:10Z)
• BiPrompt-SAM: Enhancing Image Segmentation via Explicit Selection between Point and Text Prompts [2.7218660375779513]
  BiPrompt-SAM is a novel dual-modal prompt segmentation framework.<n>It fuses spatial precision and semantic context without complex model modifications.<n>It achieves strong zero-shot performance on the Endovis17 medical dataset.
  arXiv  Detail & Related papers  (2025-03-25T15:38:55Z)
• SAMRefiner: Taming Segment Anything Model for Universal Mask Refinement [40.37217744643069]
  We propose a universal and efficient approach by adapting SAM to the mask refinement task.<n>Specifically, we introduce a multi-prompt excavation strategy to mine diverse input prompts for SAM.<n>We extend our method to SAMRefiner++ by introducing an additional IoU adaption step to further boost the performance of the generic SAMRefiner on the target dataset.
  arXiv  Detail & Related papers  (2025-02-10T18:33:15Z)
• Promptable Anomaly Segmentation with SAM Through Self-Perception Tuning [63.55145330447408]
  We propose a novel textbfSelf-textbfPerceptinon textbfTuning (textbfSPT) method for anomaly segmentation.<n>The SPT method incorporates a self-drafting tuning strategy, which generates an initial coarse draft of the anomaly mask, followed by a refinement process.
  arXiv  Detail & Related papers  (2024-11-26T08:33:25Z)
• Bridge the Points: Graph-based Few-shot Segment Anything Semantically [79.1519244940518]
  Recent advancements in pre-training techniques have enhanced the capabilities of vision foundation models. Recent studies extend the SAM to Few-shot Semantic segmentation (FSS) We propose a simple yet effective approach based on graph analysis.
  arXiv  Detail & Related papers  (2024-10-09T15:02:28Z)
• AlignSAM: Aligning Segment Anything Model to Open Context via Reinforcement Learning [61.666973416903005]
  Segment Anything Model (SAM) has demonstrated its impressive generalization capabilities in open-world scenarios with the guidance of prompts. We propose a novel framework, termed AlignSAM, designed for automatic prompting for aligning SAM to an open context.
  arXiv  Detail & Related papers  (2024-06-01T16:21:39Z)
• Stable Segment Anything Model [79.9005670886038]
  The Segment Anything Model (SAM) achieves remarkable promptable segmentation given high-quality prompts. This paper presents the first comprehensive analysis on SAM's segmentation stability across a diverse spectrum of prompt qualities. Our solution, termed Stable-SAM, offers several advantages: 1) improved SAM's segmentation stability across a wide range of prompt qualities, while 2) retaining SAM's powerful promptable segmentation efficiency and generality.
  arXiv  Detail & Related papers  (2023-11-27T12:51:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.