Do Instance Priors Help Weakly Supervised Semantic Segmentation?

• URL: http://arxiv.org/abs/2604.11170v1
• Date: Mon, 13 Apr 2026 08:29:49 GMT
• Title: Do Instance Priors Help Weakly Supervised Semantic Segmentation?
• Abstract summary: SeSAM is a framework that uses a foundational segmentation model, i.e. Segment Anything Model (SAM), with weak labels.<n>SeSAM decomposes class masks into connected components, samples point prompts along object skeletons, selects SAM masks using weak-label coverage, and iteratively refines labels using pseudo-labels.<n> integrated with a semi-supervised learning framework, SeSAM balances ground-truth labels, SAM-based pseudo-labels, and high-confidence pseudo-labels.
• Score: 82.75875820720616
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Semantic segmentation requires dense pixel-level annotations, which are costly and time-consuming to acquire. To address this, we present SeSAM, a framework that uses a foundational segmentation model, i.e. Segment Anything Model (SAM), with weak labels, including coarse masks, scribbles, and points. SAM, originally designed for instance-based segmentation, cannot be directly used for semantic segmentation tasks. In this work, we identify specific challenges faced by SAM and determine appropriate components to adapt it for class-based segmentation using weak labels. Specifically, SeSAM decomposes class masks into connected components, samples point prompts along object skeletons, selects SAM masks using weak-label coverage, and iteratively refines labels using pseudo-labels, enabling SAM-generated masks to be effectively used for semantic segmentation. Integrated with a semi-supervised learning framework, SeSAM balances ground-truth labels, SAM-based pseudo-labels, and high-confidence pseudo-labels, significantly improving segmentation quality. Extensive experiments across multiple benchmarks and weak annotation types show that SeSAM consistently outperforms weakly supervised baselines while substantially reducing annotation cost relative to fine supervision.

Related papers

• EOVSAM: Efficient Open-Vocabulary Segmentation with SAM 3 in One Pass [79.69317943751223]
  We propose an Efficient Open-Vocabulary segmentation framework with SAM 3 (EOVSAM)<n>EOVSAM adapts SAM 3 for single-pass prediction.<n>It consistently improves segmentation accuracy over vanilla SAM 3 on all evaluated datasets.
  arXiv  Detail & Related papers  (2026-08-03T14:15:37Z)
• ConformalSAM: Unlocking the Potential of Foundational Segmentation Models in Semi-Supervised Semantic Segmentation with Conformal Prediction [57.930531826380836]
  This work explores whether a foundational segmentation model can address label scarcity in the pixel-level vision task as an annotator for unlabeled images.<n>We propose ConformalSAM, a novel SSSS framework which first calibrates the foundation model using the target domain's labeled data and then filters out unreliable pixel labels of unlabeled data.
  arXiv  Detail & Related papers  (2025-07-21T17:02:57Z)
• S^4M: Boosting Semi-Supervised Instance Segmentation with SAM [25.94737539065708]
  Semi-supervised instance segmentation poses challenges due to limited labeled data.<n>Current teacher-student frameworks still suffer from performance constraints due to unreliable pseudo-label quality.
  arXiv  Detail & Related papers  (2025-04-07T17:59:10Z)
• SAM-IF: Leveraging SAM for Incremental Few-Shot Instance Segmentation [1.1141688859736805]
  We propose SAM-IF, a novel method for incremental few-shot instance segmentation leveraging the Segment Anything Model (SAM)<n>Our method achieves competitive but more reasonable results compared to existing approaches.
  arXiv  Detail & Related papers  (2024-12-15T03:11:41Z)
• SAM-CP: Marrying SAM with Composable Prompts for Versatile Segmentation [88.80792308991867]
  Segment Anything model (SAM) has shown ability to group image pixels into patches, but applying it to semantic-aware segmentation still faces major challenges.<n>This paper presents SAM-CP, a simple approach that establishes two types of composable prompts beyond SAM and composes them for versatile segmentation.<n> Experiments show that SAM-CP achieves semantic, instance, and panoptic segmentation in both open and closed domains.
  arXiv  Detail & Related papers  (2024-07-23T17:47:25Z)
• Semantic-aware SAM for Point-Prompted Instance Segmentation [29.286913777078116]
  In this paper, we introduce a cost-effective category-specific segmenter using Segment Anything (SAM) To tackle this challenge, we have devised a Semantic-Aware Instance Network (SAPNet) that integrates Multiple Instance Learning (MIL) with matching capability and SAM with point prompts. SAPNet strategically selects the most representative mask proposals generated by SAM to supervise segmentation, with a specific focus on object category information.
  arXiv  Detail & Related papers  (2023-12-26T05:56:44Z)
• Stable Segment Anything Model [79.9005670886038]
  The Segment Anything Model (SAM) achieves remarkable promptable segmentation given high-quality prompts. This paper presents the first comprehensive analysis on SAM's segmentation stability across a diverse spectrum of prompt qualities. Our solution, termed Stable-SAM, offers several advantages: 1) improved SAM's segmentation stability across a wide range of prompt qualities, while 2) retaining SAM's powerful promptable segmentation efficiency and generality.
  arXiv  Detail & Related papers  (2023-11-27T12:51:42Z)
• Segment Anything Model (SAM) Enhanced Pseudo Labels for Weakly Supervised Semantic Segmentation [30.812323329239614]
  Weakly supervised semantic segmentation (WSSS) aims to bypass the need for laborious pixel-level annotation by using only image-level annotation. Most existing methods rely on Class Activation Maps (CAM) to derive pixel-level pseudo-labels. We introduce a simple yet effective method harnessing the Segment Anything Model (SAM), a class-agnostic foundation model capable of producing fine-grained instance masks of objects, parts, and subparts.
  arXiv  Detail & Related papers  (2023-05-09T23:24:09Z)
• SegGroup: Seg-Level Supervision for 3D Instance and Semantic Segmentation [88.22349093672975]
  We design a weakly supervised point cloud segmentation algorithm that only requires clicking on one point per instance to indicate its location for annotation. With over-segmentation for pre-processing, we extend these location annotations into segments as seg-level labels. We show that our seg-level supervised method (SegGroup) achieves comparable results with the fully annotated point-level supervised methods.
  arXiv  Detail & Related papers  (2020-12-18T13:23:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.