SSP-SAM: SAM with Semantic-Spatial Prompt for Referring Expression Segmentation

• URL: http://arxiv.org/abs/2603.18086v1
• Date: Wed, 18 Mar 2026 08:26:07 GMT
• Title: SSP-SAM: SAM with Semantic-Spatial Prompt for Referring Expression Segmentation
• Abstract summary: The Segment Anything Model (SAM) excels at general image segmentation but has limited ability to understand natural language.<n>We propose SSP-SAM, a framework that fully utilizes SAM's segmentation capabilities by integrating a Semantic-Spatial Prompt (SSP) encoder.<n>SSP-SAM enhances the referent representation for the prompt generator, resulting in high-quality SSPs that enable SAM to generate precise masks guided by language.
• Score: 37.55469031169627
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The Segment Anything Model (SAM) excels at general image segmentation but has limited ability to understand natural language, which restricts its direct application in Referring Expression Segmentation (RES). Toward this end, we propose SSP-SAM, a framework that fully utilizes SAM's segmentation capabilities by integrating a Semantic-Spatial Prompt (SSP) encoder. Specifically, we incorporate both visual and linguistic attention adapters into the SSP encoder, which highlight salient objects within the visual features and discriminative phrases within the linguistic features. This design enhances the referent representation for the prompt generator, resulting in high-quality SSPs that enable SAM to generate precise masks guided by language. Although not specifically designed for Generalized RES (GRES), where the referent may correspond to zero, one, or multiple objects, SSP-SAM naturally supports this more flexible setting without additional modifications. Extensive experiments on widely used RES and GRES benchmarks confirm the superiority of our method. Notably, our approach generates segmentation masks of high quality, achieving strong precision even at strict thresholds such as Pr@0.9. Further evaluation on the PhraseCut dataset demonstrates improved performance in open-vocabulary scenarios compared to existing state-of-the-art RES methods. The code and checkpoints are available at: https://github.com/WayneTomas/SSP-SAM.

Related papers

• CLIP-Guided SAM: Parameter-Efficient Semantic Conditioning for Promptable Segmentation [6.517222960194991]
  We propose CLIP-Guided SAM, a parameter-efficient segmentation framework built on internal semantic conditioning.<n>Instead of using semantic signals only to generate prompts, we inject CLIP-derived text, vision, and similarity features directly into SAM's image encoder.<n>Our framework is designed for low labeled-data settings and applies to both general-domain benchmarks and specialized downstream tasks.
  arXiv  Detail & Related papers  (2026-05-24T01:40:30Z)
• Do Instance Priors Help Weakly Supervised Semantic Segmentation? [82.75875820720616]
  SeSAM is a framework that uses a foundational segmentation model, i.e. Segment Anything Model (SAM), with weak labels.<n>SeSAM decomposes class masks into connected components, samples point prompts along object skeletons, selects SAM masks using weak-label coverage, and iteratively refines labels using pseudo-labels.<n> integrated with a semi-supervised learning framework, SeSAM balances ground-truth labels, SAM-based pseudo-labels, and high-confidence pseudo-labels.
  arXiv  Detail & Related papers  (2026-04-13T08:29:49Z)
• Tarot-SAM3: Training-free SAM3 for Any Referring Expression Segmentation [18.568343383992072]
  We present Tarot-SAM3, a training-free framework that can accurately segment from any referring expression.<n>Tarot-SAM3 consists of two key phases. First, the Reasoning Expression Interpreter (ERI) phase introduces reasoning-assisted prompt options.<n>Second, the Mask Self-Refining (MSR) phase selects the best mask across prompt types and performs self-refinement.
  arXiv  Detail & Related papers  (2026-04-09T07:37:09Z)
• SAM3-I: Segment Anything with Instructions [86.92593395772029]
  We present SAM3-I, an enhanced framework that unifies concept-level understanding and instructionlevel reasoning within the SAM family.<n>We design a structured instruction taxonomy spanning concept, simple, and complex levels, and develop a scalable data engine to construct a dataset with diverse instruction-mask pairs.
  arXiv  Detail & Related papers  (2025-12-04T09:00:25Z)
• Unlocking the Potential of MLLMs in Referring Expression Segmentation via a Light-weight Mask Decoder [18.236863512276187]
  We propose a novel framework that exploits the inherent visual detail features encoded in the MLLM vision encoder without introducing an extra visual encoder.<n>Besides, we propose a detail-enhanced and semantic-consistent feature fusion module (DSFF) that fully integrates the detail-related visual feature with the semantic-related feature output by the large language model (LLM) of MLLM.<n>Our method generally surpasses both SAM-based and SAM-free competitors, striking a better balance between performance and cost.
  arXiv  Detail & Related papers  (2025-08-06T06:06:52Z)
• SAM-PTx: Text-Guided Fine-Tuning of SAM with Parameter-Efficient, Parallel-Text Adapters [0.5755004576310334]
  This paper introduces SAM-PTx, a parameter-efficient approach for adapting SAM using frozen CLIP-derived text embeddings as class-level semantic guidance.<n>Specifically, we propose a lightweight adapter called Parallel-Text that injects text embeddings into SAM's image, enabling semantics-guided segmentation.<n>We show that incorporating fixed text embeddings as input improves segmentation performance over purely spatial prompt baselines.
  arXiv  Detail & Related papers  (2025-07-31T23:26:39Z)
• SAM as the Guide: Mastering Pseudo-Label Refinement in Semi-Supervised Referring Expression Segmentation [66.92696817276288]
  SemiRES is a semi-supervised framework that effectively leverages a combination of labeled and unlabeled data to perform RES. SemiRES incorporates the Segment Anything Model (SAM), renowned for its precise boundary demarcation. In instances where a precise mask cannot be matched from the available candidates, we develop the Pixel-Wise Adjustment (PWA) strategy.
  arXiv  Detail & Related papers  (2024-06-03T15:42:30Z)
• PosSAM: Panoptic Open-vocabulary Segment Anything [58.72494640363136]
  PosSAM is an open-vocabulary panoptic segmentation model that unifies the strengths of the Segment Anything Model (SAM) with the vision-native CLIP model in an end-to-end framework. We introduce a Mask-Aware Selective Ensembling (MASE) algorithm that adaptively enhances the quality of generated masks and boosts the performance of open-vocabulary classification during inference for each image.
  arXiv  Detail & Related papers  (2024-03-14T17:55:03Z)
• VRP-SAM: SAM with Visual Reference Prompt [76.71829864364283]
  We propose a novel Visual Reference Prompt (VRP) encoder that empowers the Segment Anything Model (SAM)<n>VRP-SAM can utilize annotated reference images to comprehend specific objects and perform segmentation of specific objects in target image.<n>To enhance the generalization ability of VRP-SAM, the VRP encoder adopts a meta-learning strategy.
  arXiv  Detail & Related papers  (2024-02-27T17:58:09Z)
• Semantic-aware SAM for Point-Prompted Instance Segmentation [29.286913777078116]
  In this paper, we introduce a cost-effective category-specific segmenter using Segment Anything (SAM) To tackle this challenge, we have devised a Semantic-Aware Instance Network (SAPNet) that integrates Multiple Instance Learning (MIL) with matching capability and SAM with point prompts. SAPNet strategically selects the most representative mask proposals generated by SAM to supervise segmentation, with a specific focus on object category information.
  arXiv  Detail & Related papers  (2023-12-26T05:56:44Z)
• GSVA: Generalized Segmentation via Multimodal Large Language Models [72.57095903188922]
  Generalized Referring Expression (GRES) extends the scope of classic RES to refer to multiple objects in one expression or identify the empty targets absent in the image. Current solutions to GRES remain unsatisfactory since segmentation MLLMs cannot correctly handle the cases where users might reference multiple subjects in a singular prompt. We propose Generalized Vision Assistant (GSVA) to address this gap.
  arXiv  Detail & Related papers  (2023-12-15T02:54:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.