ActiveSAM: Image-Conditional Class Pruning for Fast and Accurate Open-Vocabulary Segmentation

• URL: http://arxiv.org/abs/2606.16996v1
• Date: Mon, 15 Jun 2026 17:31:30 GMT
• Title: ActiveSAM: Image-Conditional Class Pruning for Fast and Accurate Open-Vocabulary Segmentation
• Abstract summary: We introduce ActiveSAM, a training-free, zero-shot inference framework that turns SAM 3 into an active-vocabulary segmenter.<n>Across eight OVSS benchmarks, ActiveSAM improves the speed-accuracy tradeoff of training-free open-vocabulary semantic segmentation.<n>ActiveSAM is well-suited for deployment in noisy-input domains such as autonomous driving and embodied AI.
• Score: 40.432049062995326
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Segment Anything Model 3 (SAM 3) provides a strong frozen backbone for concept-prompted segmentation, but applying it directly to open-vocabulary semantic segmentation (OVSS) is inefficient: full-resolution decoding is typically run over the entire dataset vocabulary, whereas each image contains only a small active subset of classes. We introduce ActiveSAM, a training-free, zero-shot inference framework that turns SAM 3 into an active-vocabulary segmenter. ActiveSAM first canonicalizes and expands class prompts, then estimates an image-conditioned active set from a low-resolution presence preview. Only the retained classes are decoded at full resolution, using bucketed prompt multiplexing with the frozen SAM 3 decoder. The preview stage uses only class-presence evidence and skips unnecessary segmentation-head computation, while the final stage applies margin-aware background calibration to suppress low-confidence pixels. ActiveSAM requires no target-dataset training, no weight updates, and no oracle class-presence labels. Across eight OVSS benchmarks, ActiveSAM improves the speed-accuracy tradeoff of training-free open-vocabulary semantic segmentation, outperforming the current state-of-the-art SegEarth-OV3 by approximately +1.4 mIoU on average while running up to 5.5x faster on large-vocabulary datasets. ActiveSAM also demonstrates the strongest robustness under image corruption that simulates real-world distribution shift, making it well-suited for deployment in noisy-input domains such as autonomous driving and embodied AI. Code is available at https://github.com/VILA-Lab/ActiveSAM.

Related papers

• EOVSAM: Efficient Open-Vocabulary Segmentation with SAM 3 in One Pass [79.69317943751223]
  We propose an Efficient Open-Vocabulary segmentation framework with SAM 3 (EOVSAM)<n>EOVSAM adapts SAM 3 for single-pass prediction.<n>It consistently improves segmentation accuracy over vanilla SAM 3 on all evaluated datasets.
  arXiv  Detail & Related papers  (2026-08-03T14:15:37Z)
• Prompt-Calibrated SAM 3 for Open-Vocabulary Remote Sensing Semantic Segmentation [8.872901416973985]
  We propose ProC-SAM3, which calibrates SAM 3's prompt interface for remote sensing OVSS.<n>ProC-SAM3 achieves an average mIoU of 56.1%, outperforming the previous best training-free method by 3.9 percentage points.
  arXiv  Detail & Related papers  (2026-06-20T04:05:37Z)
• SegEarth-OV3: Exploring SAM 3 for Open-Vocabulary Semantic Segmentation in Remote Sensing Images [51.42466259821335]
  We present a preliminary exploration of applying SAM 3 to the remote sensing OVSS task without any training.<n>First, we implement a mask fusion strategy that combines the outputs from SAM 3's semantic segmentation head and the Transformer decoder.<n>Second, we utilize the presence score from the presence head to filter out categories that do not exist in the scene.
  arXiv  Detail & Related papers  (2025-12-09T15:42:28Z)
• FocSAM: Delving Deeply into Focused Objects in Segmenting Anything [58.042354516491024]
  The Segment Anything Model (SAM) marks a notable milestone in segmentation models. We propose FocSAM with a pipeline redesigned on two pivotal aspects. First, we propose Dynamic Window Multi-head Self-Attention (Dwin-MSA) to dynamically refocus SAM's image embeddings on the target object. Second, we propose Pixel-wise Dynamic ReLU (P-DyReLU) to enable sufficient integration of interactive information from a few initial clicks.
  arXiv  Detail & Related papers  (2024-05-29T02:34:13Z)
• WSI-SAM: Multi-resolution Segment Anything Model (SAM) for histopathology whole-slide images [8.179859593451285]
  We present WSI-SAM, enhancing Segment Anything Model (SAM) with precise object segmentation capabilities for histopathology images. To fully exploit pretrained knowledge while minimizing training overhead, we keep SAM frozen, introducing only minimal extra parameters. Our model outperforms SAM by 4.1 and 2.5 percent points on a ductal carcinoma in situ (DCIS) segmentation tasks and breast cancer metastasis segmentation task.
  arXiv  Detail & Related papers  (2024-03-14T10:30:43Z)
• EdgeSAM: Prompt-In-the-Loop Distillation for SAM [87.52687622659904]
  EdgeSAM is an accelerated variant of the Segment Anything Model (SAM)<n>Our approach involves distilling the original ViT-based SAM image encoder into a purely CNN-based architecture.<n>It is the first SAM variant that can run at over 30 FPS on an iPhone 14.
  arXiv  Detail & Related papers  (2023-12-11T18:59:52Z)
• Stable Segment Anything Model [79.9005670886038]
  The Segment Anything Model (SAM) achieves remarkable promptable segmentation given high-quality prompts. This paper presents the first comprehensive analysis on SAM's segmentation stability across a diverse spectrum of prompt qualities. Our solution, termed Stable-SAM, offers several advantages: 1) improved SAM's segmentation stability across a wide range of prompt qualities, while 2) retaining SAM's powerful promptable segmentation efficiency and generality.
  arXiv  Detail & Related papers  (2023-11-27T12:51:42Z)
• All-in-SAM: from Weak Annotation to Pixel-wise Nuclei Segmentation with Prompt-based Finetuning [16.016139980843835]
  The Segment Anything Model (SAM) is a recently proposed prompt-based segmentation model in a generic zero-shot segmentation approach. We introduce a pipeline that utilizes the SAM through the entire AI development workflow without requiring manual prompts during the inference stage. Our experimental results reveal two key findings: 1) the proposed pipeline surpasses the state-of-the-art (SOTA) methods in a nuclei segmentation task on the public Monuseg dataset, and 2) the utilization of weak and few annotations for SAM finetuning achieves competitive performance.
  arXiv  Detail & Related papers  (2023-07-01T10:12:46Z)
• How to Efficiently Adapt Large Segmentation Model(SAM) to Medical Images [15.181219203629643]
  Segment Anything (SAM) exhibits impressive capabilities in zero-shot segmentation for natural images. However, when applied to medical images, SAM suffers from noticeable performance drop. In this work, we propose to freeze SAM encoder and finetune a lightweight task-specific prediction head.
  arXiv  Detail & Related papers  (2023-06-23T18:34:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.