The Detector Teaches Itself: Lightweight Self-Supervised Adaptation for Open-Vocabulary Object Detection

• URL: http://arxiv.org/abs/2605.03642v1
• Date: Tue, 05 May 2026 11:14:30 GMT
• Title: The Detector Teaches Itself: Lightweight Self-Supervised Adaptation for Open-Vocabulary Object Detection
• Abstract summary: Open-vocabulary object detection aims to recognize objects from an open set categories, which leverages vision-language models (VLMs) pre-trained on large-scale image-text data.<n>We present Decoupled Adaptivity Training, a self-supervised fine-tuning approach to improve VLMs for cooperative model-based object detection.<n>Experiments on the COCO and LVIS datasets show that DAT consistently improves detection performance on both novel and known categories.
• Score: 8.847667302225156
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Open-vocabulary object detection aims to recognize objects from an open set of categories, which leverages vision-language models (VLMs) pre-trained on large-scale image-text data. The cooperative paradigm combines an object detector with a VLM to achieve zero-shot recognition of novel objects. However, VLMs pre-trained on full images often struggle to capture local object details, limiting their effectiveness when applied to region-level detection. We present Decoupled Adaptivity Training (DAT), a self-supervised fine-tuning approach to improve VLMs for cooperative model-based object detection. Given a cooperative model consists of a closed-set detector and a VLM, we first construct a region-aware pseudo-labeled dataset using a pre-trained closed-set object detector, in which regions corresponding to novel objects may be present but remain unlabeled or mislabeled. We then fine-tune the visual backbone of the VLM in a decoupled manner, which enhances local feature alignment while preserving global semantic knowledge via weight interpolation. DAT is a plug-and-play module that requires no inference overhead and fine-tunes less than 0.8M parameters. Experiments on the COCO and LVIS datasets show that DAT consistently improves detection performance on both novel and known categories, establishing a new state of the art in cooperative open-vocabulary detection.

Related papers

• Unveiling the Unknown: Open Vocabulary Object Detection with Scene Graphs [16.874298364235326]
  Open-vocabulary object detection seeks to identify novel object categories that were not part of the training data.<n>These methods often overlook structured, image-specific relationships between objects, such as interactions and spatial arrangements.<n>We propose a Scene-guided Modeling detection framework to address this issue.
  arXiv  Detail & Related papers  (2026-06-04T09:23:08Z)
• Zero-shot HOI Detection with MLLM-based Detector-agnostic Interaction Recognition [71.5328300638085]
  Zero-shot Human-object interaction (HOI) detection aims to locate humans and objects in images and recognize their interactions.<n>Existing methods, including two-stage methods, tightly couple interaction recognition with a specific detector.<n>We propose a decoupled framework that separates object detection from IR and leverages multi-modal large language models (MLLMs) for zero-shot IR.
  arXiv  Detail & Related papers  (2026-02-16T19:01:31Z)
• OW-Rep: Open World Object Detection with Instance Representation Learning [1.8749305679160366]
  Open World Object Detection (OWOD) addresses realistic scenarios where unseen object classes emerge.<n>We extend the OWOD framework to jointly detect unknown objects and learn semantically rich instance embeddings.
  arXiv  Detail & Related papers  (2024-09-24T13:13:34Z)
• Weakly Supervised Open-Vocabulary Object Detection [31.605276665964787]
  We propose a novel weakly supervised open-vocabulary object detection framework, namely WSOVOD, to extend traditional WSOD. To achieve this, we explore three vital strategies, including dataset-level feature adaptation, image-level salient object localization, and region-level vision-language alignment.
  arXiv  Detail & Related papers  (2023-12-19T18:59:53Z)
• Simple Image-level Classification Improves Open-vocabulary Object Detection [27.131298903486474]
  Open-Vocabulary Object Detection (OVOD) aims to detect novel objects beyond a given set of base categories on which the detection model is trained. Recent OVOD methods focus on adapting the image-level pre-trained vision-language models (VLMs), such as CLIP, to a region-level object detection task via, eg., region-level knowledge distillation, regional prompt learning, or region-text pre-training. These methods have demonstrated remarkable performance in recognizing regional visual concepts, but they are weak in exploiting the VLMs' powerful global scene understanding ability learned from the billion-scale
  arXiv  Detail & Related papers  (2023-12-16T13:06:15Z)
• Enhancing Novel Object Detection via Cooperative Foundational Models [68.93124785575739]
  We present a novel approach to transform existing closed-set detectors into open-set detectors.<n>We surpass the current state-of-the-art by a margin of 7.2 $ textAP_50 $ for novel classes.
  arXiv  Detail & Related papers  (2023-11-19T17:28:28Z)
• Weakly-supervised Contrastive Learning for Unsupervised Object Discovery [52.696041556640516]
  Unsupervised object discovery is promising due to its ability to discover objects in a generic manner. We design a semantic-guided self-supervised learning model to extract high-level semantic features from images. We introduce Principal Component Analysis (PCA) to localize object regions.
  arXiv  Detail & Related papers  (2023-07-07T04:03:48Z)
• Unified Visual Relationship Detection with Vision and Language Models [89.77838890788638]
  This work focuses on training a single visual relationship detector predicting over the union of label spaces from multiple datasets. We propose UniVRD, a novel bottom-up method for Unified Visual Relationship Detection by leveraging vision and language models. Empirical results on both human-object interaction detection and scene-graph generation demonstrate the competitive performance of our model.
  arXiv  Detail & Related papers  (2023-03-16T00:06:28Z)
• Learning Open-World Object Proposals without Learning to Classify [110.30191531975804]
  We propose a classification-free Object Localization Network (OLN) which estimates the objectness of each region purely by how well the location and shape of a region overlaps with any ground-truth object. This simple strategy learns generalizable objectness and outperforms existing proposals on cross-category generalization.
  arXiv  Detail & Related papers  (2021-08-15T14:36:02Z)
• Pairwise Similarity Knowledge Transfer for Weakly Supervised Object Localization [53.99850033746663]
  We study the problem of learning localization model on target classes with weakly supervised image labels. In this work, we argue that learning only an objectness function is a weak form of knowledge transfer. Experiments on the COCO and ILSVRC 2013 detection datasets show that the performance of the localization model improves significantly with the inclusion of pairwise similarity function.
  arXiv  Detail & Related papers  (2020-03-18T17:53:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.