VL-SAM-v3: Memory-Guided Visual Priors for Open-World Object Detection

• URL: http://arxiv.org/abs/2605.03456v3
• Date: Sat, 09 May 2026 04:48:21 GMT
• Title: VL-SAM-v3: Memory-Guided Visual Priors for Open-World Object Detection
• Abstract summary: We propose a unified framework that augments open-world detection with retrieval-grounded external visual memory.<n> VL-SAM-v3 consistently improves detection performance under both open-vocabulary and open-ended inference.<n>Experiments with a stronger open-vocabulary detector (i.e., SAM3) validate the generality of the proposed retrieval-and-refinement mechanism.
• Score: 16.02875655103583
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Open-world object detection aims to localize and recognize objects beyond a fixed closed-set label space. It is commonly divided into two categories, i.e., open-vocabulary detection, which assumes a predefined category list at test time, and open-ended detection, which requires generating candidate categories during the inference. Existing methods rely primarily on coarse textual semantics and parametric knowledge, which often provide insufficient visual evidence for fine-grained appearance variation, rare categories, and cluttered scenes. In this paper, we propose VL-SAM-v3, a unified framework that augments open-world detection with retrieval-grounded external visual memory. Specifically, once candidate categories are available, VL-SAM-v3 retrieves relevant visual prototypes from a non-parametric memory bank and transforms them into two complementary visual priors, i.e., sparse priors for instance-level spatial anchoring and dense priors for class-aware local context. These priors are integrated with the original detection prompts via Memory-Guided Prompt Refinement, enabling a shared retrieval-and-refinement mechanism that supports open-vocabulary and open-ended inference. Extensive zero-shot experiments on LVIS show that VL-SAM-v3 consistently improves detection performance under both open-vocabulary and open-ended inference, with particularly strong gains on rare categories. Moreover, experiments with a stronger open-vocabulary detector (i.e., SAM3) validate the generality of the proposed retrieval-and-refinement mechanism.

Related papers

• VocaDet: Sample-Driven Open-Vocabulary Object Detection and Segmentation via Visual Tokenization and Vector Database Retrieval [4.484170173286332]
  We propose VocaDet, a sample-driven open-vocabulary object detection and segmentation framework.<n>We transform continuous visual representations into discrete visual vocabularies and perform efficient retrieval-based recognition through a scalable vector database.<n> Experiments on the UA-DETRAC dataset demonstrate that VocaDet achieves effective open-vocabulary detection performance without conventional detector training.
  arXiv  Detail & Related papers  (2026-07-09T14:35:37Z)
• Beyond Open Vocabulary: Multimodal Prompting for Object Detection in Remote Sensing Images [52.7196029918473]
  Open-vocabulary object detection in remote sensing commonly relies on text-only prompting to specify target categories.<n>In practice, this assumption often breaks down in remote sensing scenarios due to task- and application-specific category semantics.<n>We propose RS-MPOD, a multimodal open-vocabulary detection framework that reformulates category specification beyond text-only prompting.
  arXiv  Detail & Related papers  (2026-02-02T11:03:01Z)
• WeDetect: Fast Open-Vocabulary Object Detection as Retrieval [74.39703419628829]
  Open-vocabulary object detection aims to detect arbitrary classes via text prompts.<n> Methods without cross-modal fusion layers (non-fusion) offer faster inference by treating recognition as a retrieval problem.<n>We develop a model family named WeDetect to achieve state-of-the-art performance across 15 benchmarks with high inference efficiency.
  arXiv  Detail & Related papers  (2025-12-13T12:40:28Z)
• VL-SAM-V2: Open-World Object Detection with General and Specific Query Fusion [7.719330752075467]
  We present an open-world object detection framework capable of discovering unseen objects while achieving favorable performance.<n>By adjusting queries from open-set models, we enable VL-SAM-V2 to be evaluated in the open-set or open-ended mode.<n> Experimental results on LVIS show that our method surpasses the previous open-set and open-ended methods, especially on rare objects.
  arXiv  Detail & Related papers  (2025-05-25T05:44:02Z)
• Comprehensive Multi-Modal Prototypes are Simple and Effective Classifiers for Vast-Vocabulary Object Detection [68.26282316080558]
  Current open-world detectors can recognize a broader range of vocabularies, despite being trained on limited categories.<n>We introduce Prova, a prototype classifier for vast-vocabulary object detection.
  arXiv  Detail & Related papers  (2024-12-23T18:57:43Z)
• LaMI-DETR: Open-Vocabulary Detection with Language Model Instruction [63.668635390907575]
  Existing methods enhance open-vocabulary object detection by leveraging the robust open-vocabulary recognition capabilities of Vision-Language Models (VLMs) We propose the Language Model Instruction (LaMI) strategy, which leverages the relationships between visual concepts and applies them within a simple yet effective DETR-like detector.
  arXiv  Detail & Related papers  (2024-07-16T02:58:33Z)
• DetCLIPv3: Towards Versatile Generative Open-vocabulary Object Detection [111.68263493302499]
  We introduce DetCLIPv3, a high-performing detector that excels at both open-vocabulary object detection and hierarchical labels. DetCLIPv3 is characterized by three core designs: 1) Versatile model architecture; 2) High information density data; and 3) Efficient training strategy. DetCLIPv3 demonstrates superior open-vocabulary detection performance, outperforming GLIPv2, GroundingDINO, and DetCLIPv2 by 18.0/19.6/6.6 AP, respectively.
  arXiv  Detail & Related papers  (2024-04-14T11:01:44Z)
• Weakly Supervised Open-Vocabulary Object Detection [31.605276665964787]
  We propose a novel weakly supervised open-vocabulary object detection framework, namely WSOVOD, to extend traditional WSOD. To achieve this, we explore three vital strategies, including dataset-level feature adaptation, image-level salient object localization, and region-level vision-language alignment.
  arXiv  Detail & Related papers  (2023-12-19T18:59:53Z)
• Multi-Modal Classifiers for Open-Vocabulary Object Detection [104.77331131447541]
  The goal of this paper is open-vocabulary object detection (OVOD) We adopt a standard two-stage object detector architecture. We explore three ways via: language descriptions, image exemplars, or a combination of the two.
  arXiv  Detail & Related papers  (2023-06-08T18:31:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.