OVS-DINO: Open-Vocabulary Segmentation via Structure-Aligned SAM-DINO with Language Guidance

• URL: http://arxiv.org/abs/2604.08461v1
• Date: Thu, 09 Apr 2026 16:57:11 GMT
• Title: OVS-DINO: Open-Vocabulary Segmentation via Structure-Aligned SAM-DINO with Language Guidance
• Abstract summary: Open-Vocabulary aims to segment image regions beyond predefined category sets by leveraging semantic descriptions.<n>Recent efforts have incorporated Vision Foundation Models (VFMs) like DINO to alleviate these limitations.<n>We propose OVS-DINO, a novel framework that revitalizes latent edge-sensitivity of DINO through structural alignment.
• Score: 53.45125687428705
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Open-Vocabulary Segmentation (OVS) aims to segment image regions beyond predefined category sets by leveraging semantic descriptions. While CLIP based approaches excel in semantic generalization, they frequently lack the fine-grained spatial awareness required for dense prediction. Recent efforts have incorporated Vision Foundation Models (VFMs) like DINO to alleviate these limitations. However, these methods still struggle with the precise edge perception necessary for high fidelity segmentation. In this paper, we analyze internal representations of DINO and discover that its inherent boundary awareness is not absent but rather undergoes progressive attenuation as features transition into deeper transformer blocks. To address this, we propose OVS-DINO, a novel framework that revitalizes latent edge-sensitivity of DINO through structural alignment with the Segment Anything Model (SAM). Specifically, we introduce a Structure-Aware Encoder (SAE) and a Structure-Modulated Decoder (SMD) to effectively activate boundary features of DINO using SAM's structural priors, complemented by a supervision strategy utilizing SAM generated pseudo-masks. Extensive experiments demonstrate that our method achieves state-of-the-art performance across multiple weakly-supervised OVS benchmarks, improving the average score by 2.1% (from 44.8% to 46.9%). Notably, our approach significantly enhances segmentation accuracy in complex, cluttered scenarios, with a gain of 6.3% on Cityscapes (from 36.6% to 42.9%).

Related papers

• Bridging Vision Foundation Model Priors with CLIP for Spatial-aware Few-shot Anomaly Detection in Medical Images [48.646448077284276]
  We propose Spatial-FAD, a framework for effective few-shot medical anomaly detection.<n>It improves lesion localization by combining VFM spatial priors with CLIP semantics.<n>Our method achieves an average improvement of over 11.4% in Dice score and 1.8% in AUC.
  arXiv  Detail & Related papers  (2026-09-11T05:21:01Z)
• Rethinking Attention Locality in Spiking Transformers [50.00501275379363]
  Spiking Transformers provide a promising paradigm for efficient visual processing with spike-driven computation.<n>Their Softmax-free Spiking Self-Attention (SSA) struggles to establish spatially localized token interactions.<n>We propose Spatially Contiguous Local Attention with Boundary Continuity Pathway (SCLA-BCP)<n>Our approach improves mAP@50 by up to 9.50% on COCO on 2017 and mIoU by up to 3.42% on ADE20K.
  arXiv  Detail & Related papers  (2026-08-09T07:36:17Z)
• EOVSAM: Efficient Open-Vocabulary Segmentation with SAM 3 in One Pass [79.69317943751223]
  We propose an Efficient Open-Vocabulary segmentation framework with SAM 3 (EOVSAM)<n>EOVSAM adapts SAM 3 for single-pass prediction.<n>It consistently improves segmentation accuracy over vanilla SAM 3 on all evaluated datasets.
  arXiv  Detail & Related papers  (2026-08-03T14:15:37Z)
• Step-Attention Refinement of DINOv3 Features for Efficient Anterior Eye Segmentation [0.0]
  Anterior eye segment (AES) segmentation is a key component of both ocular biometrics and emerging clinical image analysis applications.<n>We investigate robust AES segmentation in clinical settings, and propose a lightweight architecture built upon a distilled DINOv3 ViT-Small backbone.<n>We evaluate the proposed approach on a private dataset of 333 clinically acquired AES images spanning eight ophthalmic acquisition protocols and annotated for seven anatomical classes.
  arXiv  Detail & Related papers  (2026-07-29T16:11:11Z)
• Decouple and Rectify: Semantics-Preserving Structural Enhancement for Open-Vocabulary Remote Sensing Segmentation [23.298715255853782]
  Open-vocabulary semantic segmentation in the remote sensing (RS) field requires both language-aligned recognition and fine-grained spatial delineation.<n>Recent methods attempt to compensate for this by introducing RS-pretrained DINO features.<n>We propose DR-Seg, a novel de-couple-and-rectify framework.
  arXiv  Detail & Related papers  (2026-04-02T13:15:05Z)
• SSVP: Synergistic Semantic-Visual Prompting for Industrial Zero-Shot Anomaly Detection [55.54007781679915]
  We propose Synergistic Semantic-Visual Prompting (SSVP), that efficiently fuses diverse visual encodings to elevate model's fine-grained perception.<n>SSVP achieves state-of-the-art performance with 93.0% Image-AUROC and 92.2% Pixel-AUROC on MVTec-AD, significantly outperforming existing zero-shot approaches.
  arXiv  Detail & Related papers  (2026-01-14T04:42:19Z)
• SAM-Aug: Leveraging SAM Priors for Few-Shot Parcel Segmentation in Satellite Time Series [3.4368348203064283]
  We propose SAM-Aug, a new annotation-efficient framework to improve few-shot land cover mapping.<n>Our approach constructs cloud-free composite images from temporal sequences and applies SAM in a fully unsupervised manner.<n>Experiments on the PASTIS-R benchmark under a 5 percent labeled setting demonstrate the effectiveness and robustness of SAM-Aug.
  arXiv  Detail & Related papers  (2026-01-14T03:18:04Z)
• Structure-Aware Feature Rectification with Region Adjacency Graphs for Training-Free Open-Vocabulary Semantic Segmentation [22.409969687852506]
  We propose a structure-aware feature rectification approach that incorporates instance-specific priors derived directly from the image.<n>Our method effectively suppresses segmentation noise, improves region-level consistency, and achieves strong performance on multiple open-vocabulary segmentation benchmarks.
  arXiv  Detail & Related papers  (2025-12-08T10:00:36Z)
• VesSAM: Efficient Multi-Prompting for Segmenting Complex Vessel [68.24765319399286]
  We present VesSAM, a powerful and efficient framework tailored for 2D vessel segmentation.<n>VesSAM integrates (1) a convolutional adapter to enhance local texture features, (2) a multi-prompt encoder that fuses anatomical prompts, and (3) a lightweight mask decoder to reduce jagged artifacts.<n>VesSAM consistently outperforms state-of-the-art PEFT-based SAM variants by over 10% Dice and 13% IoU.
  arXiv  Detail & Related papers  (2025-11-02T15:47:05Z)
• Harnessing Vision Foundation Models for High-Performance, Training-Free Open Vocabulary Segmentation [26.786890883280062]
  We introduce Trident, a training-free framework that first splices features extracted by CLIP and DINO from sub-images, then leverages SAM's encoder to create a correlation matrix for global aggregation. Trident achieves a significant improvement in the mIoU across eight benchmarks compared with the current SOTA, increasing from 44.4 to 48.6.Code.
  arXiv  Detail & Related papers  (2024-11-14T06:31:20Z)
• PosSAM: Panoptic Open-vocabulary Segment Anything [58.72494640363136]
  PosSAM is an open-vocabulary panoptic segmentation model that unifies the strengths of the Segment Anything Model (SAM) with the vision-native CLIP model in an end-to-end framework. We introduce a Mask-Aware Selective Ensembling (MASE) algorithm that adaptively enhances the quality of generated masks and boosts the performance of open-vocabulary classification during inference for each image.
  arXiv  Detail & Related papers  (2024-03-14T17:55:03Z)
• ISTR: End-to-End Instance Segmentation with Transformers [147.14073165997846]
  We propose an instance segmentation Transformer, termed ISTR, which is the first end-to-end framework of its kind. ISTR predicts low-dimensional mask embeddings, and matches them with ground truth mask embeddings for the set loss. Benefiting from the proposed end-to-end mechanism, ISTR demonstrates state-of-the-art performance even with approximation-based suboptimal embeddings.
  arXiv  Detail & Related papers  (2021-05-03T06:00:09Z)
• On the Structures of Representation for the Robustness of Semantic Segmentation to Input Corruption [16.897599251748982]
  Implicit Background Estimation (IBE) has demonstrated to be a promising technique to improve the robustness to out-of-distribution inputs for semantic segmentation models for little to no cost. In this paper, we provide analysis comparing the structures learned as a result of optimization objectives that use Softmax, IBE, and Sigmoid. We demonstrate that SCrIBE exhibits superior segmentation performance aggregated across all corruptions and severity levels with a mIOU of 42.1 compared to both IBE 40.3 and the Softmax Baseline 37.5.
  arXiv  Detail & Related papers  (2020-09-02T04:49:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.