SegRAG: Training-Free Retrieval-Augmented Semantic Segmentation

• URL: http://arxiv.org/abs/2605.17630v2
• Date: Tue, 19 May 2026 18:36:14 GMT
• Title: SegRAG: Training-Free Retrieval-Augmented Semantic Segmentation
• Abstract summary: SegRAG is a training-free retrieval-augmented segmentation framework.<n>It grounds SAM3 with class-specific point prompts derived from a curated DINOv3 feature bank.<n>On four standard benchmarks, SegRAG consistently outperforms the text-only baseline.
• Score: 13.665861251747144
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Open-vocabulary segmentation models such as SAM3 perform well across broad categories via text prompting, yet degrade when target classes are visually underrepresented in pretraining or depart from canonical depictions-limitations text prompts cannot resolve spatially. We present SegRAG, a training-free retrieval-augmented segmentation framework that grounds SAM3 with class-specific point prompts derived from a curated DINOv3 feature bank. Offline, dense patch-level descriptors are extracted from annotated references and filtered by Intra-Class Cohesion Distillation (ICCD), retaining only prototypes that reliably retrieve within-class foreground. At inference, Topographic Similarity Grounding (TSG) computes a cosine-similarity landscape against retrieved prototypes, identifies coherent high-confidence regions via connected-component analysis, and extracts peak locations through non-maximum suppression. The resulting point prompts are delivered jointly with class-name text in a single SAM3 forward pass. On four standard benchmarks, SegRAG consistently outperforms the text-only baseline, gaining up to +3.92 mIoU on LVIS. On AgML agricultural benchmarks under zero-shot domain transfer, it raises mean IoU from 25.27 to 59.24 (+33.97) and recovers individual classes from zero to over 95 mIoU. Ablations confirm that ICCD, TSG, and joint prompting each contribute independently and compound when combined. Code is available at (https://github.com/boudiafA/SegRAG).

Related papers

• SeGDeP: Semantic- and Geometric-Aware Decoupled Prompts for Reasoning Segmentation [51.46828526392219]
  Reasoning segmentation converts an implicit linguistic conclusion into a precise mask.<n>Existing MLLM-segmenter interfaces either use a special trigger or compress both signals into one context.<n>We present SeGDeP, an explicit what-where interface.
  arXiv  Detail & Related papers  (2026-09-08T15:11:32Z)
• Structuring Semantic Embeddings for Principle Evaluation: A Prototype-Guided Contrastive Learning Approach [58.15427952222424]
  General-purpose text embeddings are widely deployed for such tasks, but broad semantic similarity can place semantically similar yet task-distinct examples.<n>We introduce Prototype-Guided Contrastive Learning (PGCL), a prototype-guided geometric regularization module built on top of frozen text embeddings.
  arXiv  Detail & Related papers  (2026-08-15T13:19:25Z)
• Prompt-Calibrated SAM 3 for Open-Vocabulary Remote Sensing Semantic Segmentation [8.872901416973985]
  We propose ProC-SAM3, which calibrates SAM 3's prompt interface for remote sensing OVSS.<n>ProC-SAM3 achieves an average mIoU of 56.1%, outperforming the previous best training-free method by 3.9 percentage points.
  arXiv  Detail & Related papers  (2026-06-20T04:05:37Z)
• ET-SAM: Efficient Point Prompt Prediction in SAM for Unified Scene Text Detection and Layout Analysis [39.062952450992746]
  ET-SAM is an Efficient framework with two decoders for unified scene Text detection and layout analysis based on SAM.<n>We customize a lightweight point decoder that produces word heatmaps for achieving a few foreground points.<n>We introduce three corresponding sets of learnable task prompts in both the point decoder and hierarchical mask decoder to mitigate discrepancies across datasets.
  arXiv  Detail & Related papers  (2026-03-26T08:37:32Z)
• SegEarth-OV3: Exploring SAM 3 for Open-Vocabulary Semantic Segmentation in Remote Sensing Images [51.42466259821335]
  We present a preliminary exploration of applying SAM 3 to the remote sensing OVSS task without any training.<n>First, we implement a mask fusion strategy that combines the outputs from SAM 3's semantic segmentation head and the Transformer decoder.<n>Second, we utilize the presence score from the presence head to filter out categories that do not exist in the scene.
  arXiv  Detail & Related papers  (2025-12-09T15:42:28Z)
• Exploring Efficient Open-Vocabulary Segmentation in the Remote Sensing [55.291219073365546]
  Open-Vocabulary Remote Sensing Image (OVRSIS) is an emerging task that adapts Open-Vocabulary (OVS) to the remote sensing (RS) domain.<n>textbfRSKT-Seg is a novel open-vocabulary segmentation framework tailored for remote sensing.<n> RSKT-Seg consistently outperforms strong OVS baselines by +3.8 mIoU and +5.9 mACC, while achieving 2x faster inference through efficient aggregation.
  arXiv  Detail & Related papers  (2025-09-15T15:24:49Z)
• Open-world Point Cloud Semantic Segmentation: A Human-in-the-loop Framework [8.451270206964534]
  Open-world point cloud semantic segmentation (OW-Seg) aims to predict point labels of both base and novel classes in real-world scenarios.<n>We propose HOW-Seg, the first human-in-the-loop framework for OW-Seg.<n>By leveraging sparse human annotations as guidance, HOW-Seg enables prototype-based segmentation for both base and novel classes.
  arXiv  Detail & Related papers  (2025-08-07T01:20:41Z)
• RG-SAN: Rule-Guided Spatial Awareness Network for End-to-End 3D Referring Expression Segmentation [72.95147072227998]
  3D Referring Expression aims to segment 3D objects by correlating referring expressions with point clouds.<n>Traditional approaches frequently encounter issues like over-segmentation or mis-segmentation, due to insufficient emphasis on spatial information of instances.<n>We introduce a Rule-Guided Spatial Awareness Network (RG-SAN) by utilizing solely the spatial information of the target instance for supervision.
  arXiv  Detail & Related papers  (2024-12-03T11:50:16Z)
• LESS: Label-Efficient and Single-Stage Referring 3D Segmentation [55.06002976797879]
  Referring 3D is a visual-language task that segments all points of the specified object from a 3D point cloud described by a sentence of query. We propose a novel Referring 3D pipeline, Label-Efficient and Single-Stage, dubbed LESS, which is only under the supervision of efficient binary mask. We achieve state-of-the-art performance on ScanRefer dataset by surpassing the previous methods about 3.7% mIoU using only binary labels.
  arXiv  Detail & Related papers  (2024-10-17T07:47:41Z)
• Subspace Prototype Guidance for Mitigating Class Imbalance in Point Cloud Semantic Segmentation [23.250178208474928]
  This paper introduces a novel method, namely subspace prototype guidance (textbfSPG) to guide the training of segmentation network. The proposed method significantly improves the segmentation performance and surpasses the state-of-the-art method.
  arXiv  Detail & Related papers  (2024-08-20T04:31:46Z)
• SegVG: Transferring Object Bounding Box to Segmentation for Visual Grounding [56.079013202051094]
  We present SegVG, a novel method transfers the box-level annotation as signals to provide an additional pixel-level supervision for Visual Grounding. This approach allows us to iteratively exploit the annotation as signals for both box-level regression and pixel-level segmentation.
  arXiv  Detail & Related papers  (2024-07-03T15:30:45Z)
• A Language-Guided Benchmark for Weakly Supervised Open Vocabulary Semantic Segmentation [10.054960979867584]
  We propose a novel weakly supervised OVSS pipeline that can perform ZSS, FSS and Cross-dataset segmentation on novel classes. The proposed pipeline beats existing methods for weak generalized Zero-Shot and weak Few-Shot semantic segmentation by 39 and 3 mIOU points respectively.
  arXiv  Detail & Related papers  (2023-02-27T21:55:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.