VocaDet: Sample-Driven Open-Vocabulary Object Detection and Segmentation via Visual Tokenization and Vector Database Retrieval

• URL: http://arxiv.org/abs/2607.08541v1
• Date: Thu, 09 Jul 2026 14:35:37 GMT
• Title: VocaDet: Sample-Driven Open-Vocabulary Object Detection and Segmentation via Visual Tokenization and Vector Database Retrieval
• Abstract summary: We propose VocaDet, a sample-driven open-vocabulary object detection and segmentation framework.<n>We transform continuous visual representations into discrete visual vocabularies and perform efficient retrieval-based recognition through a scalable vector database.<n> Experiments on the UA-DETRAC dataset demonstrate that VocaDet achieves effective open-vocabulary detection performance without conventional detector training.
• Score: 4.484170173286332
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Open-vocabulary object detection and segmentation aim to recognize arbitrary objects beyond predefined categories. Although recent vision-language and reference-based approaches have significantly advanced this field, they often rely on text prompts, limited visual examples, or expensive feature matching procedures, making them difficult to scale to large and continuously expanding object repositories. In this work, we propose VocaDet, a sample-driven open-vocabulary object detection and segmentation framework that learns object concepts directly from user-provided positive and negative sample collections without model retraining. The key idea is to transform continuous visual representations into discrete visual vocabularies and perform efficient retrieval-based recognition through a scalable vector database. Specifically, we employ DINOv3 as the visual feature extractor and apply agglomerative clustering with adaptive clustering sensitivity to generate multi-granularity visual tokens. These visual tokens, together with position-debiased representations and spatial topology information, are stored as expandable object memories in a vector database. During inference, query images are converted into visual tokens and efficiently matched against the stored object memories for object localization and segmentation. Furthermore, a background filtering mechanism is introduced to remove frequently occurring background patterns and reduce redundant retrieval operations in practical fixed-camera scenarios. Experiments on the UA-DETRAC dataset demonstrate that VocaDet achieves effective open-vocabulary detection performance without conventional detector training, while supporting continuously expandable recognition capability as additional positive and negative samples are accumulated.

Related papers

• Revisiting Human-in-the-Loop Object Retrieval with Pre-Trained Vision Transformers [1.8631279231581521]
  Human-in-the-Loop Object Retrieval is a task that consists of iteratively retrieving images containing objects of a class-of-interest.<n>We revisit the task by leveraging pre-trained ViT representations and addressing key design questions.
  arXiv  Detail & Related papers  (2026-04-01T12:18:17Z)
• WeDetect: Fast Open-Vocabulary Object Detection as Retrieval [74.39703419628829]
  Open-vocabulary object detection aims to detect arbitrary classes via text prompts.<n> Methods without cross-modal fusion layers (non-fusion) offer faster inference by treating recognition as a retrieval problem.<n>We develop a model family named WeDetect to achieve state-of-the-art performance across 15 benchmarks with high inference efficiency.
  arXiv  Detail & Related papers  (2025-12-13T12:40:28Z)
• Composed Object Retrieval: Object-level Retrieval via Composed Expressions [71.47650333199628]
  Composed Object Retrieval (COR) is a brand-new task that goes beyond image-level retrieval to achieve object-level precision.<n>We construct COR127K, the first large-scale COR benchmark that contains 127,166 retrieval triplets with various semantic transformations in 408 categories.<n>We also present CORE, a unified end-to-end model that integrates reference region encoding, adaptive visual-textual interaction, and region-level contrastive learning.
  arXiv  Detail & Related papers  (2025-08-06T13:11:40Z)
• From Open-Vocabulary to Vocabulary-Free Semantic Segmentation [78.62232202171919]
  Open-vocabulary semantic segmentation enables models to identify novel object categories beyond their training data.<n>Current approaches still rely on manually specified class names as input, creating an inherent bottleneck in real-world applications.<n>This work proposes a Vocabulary-Free Semantic pipeline, eliminating the need for predefined class vocabularies.
  arXiv  Detail & Related papers  (2025-02-17T15:17:08Z)
• RSRefSeg: Referring Remote Sensing Image Segmentation with Foundation Models [24.67117013862316]
  Referring remote sensing image segmentation is crucial for achieving fine-grained visual understanding.<n>We introduce a referring remote sensing image segmentation foundational model, RSRefSeg.<n> Experimental results on the RRSIS-D dataset demonstrate that RSRefSeg outperforms existing methods.
  arXiv  Detail & Related papers  (2025-01-12T13:22:35Z)
• Distilling Spectral Graph for Object-Context Aware Open-Vocabulary Semantic Segmentation [47.047267066525265]
  We introduce a novel approach that incorporates object-level contextual knowledge within images.<n>Our proposed approach achieves state-of-the-art performance with strong generalizability across diverse datasets.
  arXiv  Detail & Related papers  (2024-11-26T06:34:48Z)
• Teaching VLMs to Localize Specific Objects from In-context Examples [56.797110842152]
  We find that present-day Vision-Language Models (VLMs) lack a fundamental cognitive ability: learning to localize specific objects in a scene by taking into account the context.<n>This work is the first to explore and benchmark personalized few-shot localization for VLMs.
  arXiv  Detail & Related papers  (2024-11-20T13:34:22Z)
• Boosting Open-Vocabulary Object Detection by Handling Background Samples [9.07525578809556]
  We propose a novel approach to address the limitations of CLIP in handling background samples. We introduce Partial Object Suppression (POS) to address the issue of misclassifying partial regions as foreground. Our proposed model is capable of achieving performance enhancements across various open-vocabulary detectors.
  arXiv  Detail & Related papers  (2024-10-11T09:15:50Z)
• Generative Region-Language Pretraining for Open-Ended Object Detection [55.42484781608621]
  We propose a framework named GenerateU, which can detect dense objects and generate their names in a free-form way. Our framework achieves comparable results to the open-vocabulary object detection method GLIP.
  arXiv  Detail & Related papers  (2024-03-15T10:52:39Z)
• Open-Vocabulary Camouflaged Object Segmentation [66.94945066779988]
  We introduce a new task, open-vocabulary camouflaged object segmentation (OVCOS) We construct a large-scale complex scene dataset (textbfOVCamo) containing 11,483 hand-selected images with fine annotations and corresponding object classes. By integrating the guidance of class semantic knowledge and the supplement of visual structure cues from the edge and depth information, the proposed method can efficiently capture camouflaged objects.
  arXiv  Detail & Related papers  (2023-11-19T06:00:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.