Open-vocabulary vs. Closed-set: Best Practice for Few-shot Object Detection Considering Text Describability

• URL: http://arxiv.org/abs/2410.15315v1
• Date: Sun, 20 Oct 2024 06:59:35 GMT
• Title: Open-vocabulary vs. Closed-set: Best Practice for Few-shot Object Detection Considering Text Describability
• Authors: Yusuke Hosoya, Masanori Suganuma, Takayuki Okatani,
• Abstract summary: In real-world applications, the target class concepts is often hard to describe in text. There is a high demand for few-shot object detection (FSOD) Can the benefits of OVD extend to FSOD for object classes that are difficult to describe in text?
• Score: 19.54008511592332
• License:
• Abstract: Open-vocabulary object detection (OVD), detecting specific classes of objects using only their linguistic descriptions (e.g., class names) without any image samples, has garnered significant attention. However, in real-world applications, the target class concepts is often hard to describe in text and the only way to specify target objects is to provide their image examples, yet it is often challenging to obtain a good number of samples. Thus, there is a high demand from practitioners for few-shot object detection (FSOD). A natural question arises: Can the benefits of OVD extend to FSOD for object classes that are difficult to describe in text? Compared to traditional methods that learn only predefined classes (referred to in this paper as closed-set object detection, COD), can the extra cost of OVD be justified? To answer these questions, we propose a method to quantify the ``text-describability'' of object detection datasets using the zero-shot image classification accuracy with CLIP. This allows us to categorize various OD datasets with different text-describability and emprically evaluate the FSOD performance of OVD and COD methods within each category. Our findings reveal that: i) there is little difference between OVD and COD for object classes with low text-describability under equal conditions in OD pretraining; and ii) although OVD can learn from more diverse data than OD-specific data, thereby increasing the volume of training data, it can be counterproductive for classes with low-text-describability. These findings provide practitioners with valuable guidance amidst the recent advancements of OVD methods.

Related papers

• Exploiting Unlabeled Data with Multiple Expert Teachers for Open Vocabulary Aerial Object Detection and Its Orientation Adaptation [58.37525311718006]
  We put forth a novel formulation of the aerial object detection problem, namely open-vocabulary aerial object detection (OVAD) We propose CastDet, a CLIP-activated student-teacher detection framework that serves as the first OVAD detector specifically designed for the challenging aerial scenario. Our framework integrates a robust localization teacher along with several box selection strategies to generate high-quality proposals for novel objects.
  arXiv  Detail & Related papers  (2024-11-04T12:59:13Z)
• Beyond Few-shot Object Detection: A Detailed Survey [25.465534270637523]
  Researchers have introduced few-shot object detection (FSOD) approaches that merge few-shot learning and object detection principles. These approaches play a vital role in reducing the reliance on extensive labeled datasets. This survey paper aims to provide a comprehensive understanding of the above-mentioned few-shot settings and explore the methodologies for each FSOD task.
  arXiv  Detail & Related papers  (2024-08-26T13:09:23Z)
• Learning Background Prompts to Discover Implicit Knowledge for Open Vocabulary Object Detection [101.15777242546649]
  Open vocabulary object detection (OVD) aims at seeking an optimal object detector capable of recognizing objects from both base and novel categories. Recent advances leverage knowledge distillation to transfer insightful knowledge from pre-trained large-scale vision-language models to the task of object detection. We present a novel OVD framework termed LBP to propose learning background prompts to harness explored implicit background knowledge.
  arXiv  Detail & Related papers  (2024-06-01T17:32:26Z)
• The devil is in the fine-grained details: Evaluating open-vocabulary object detectors for fine-grained understanding [8.448399308205266]
  We introduce an evaluation protocol based on dynamic vocabulary generation to test whether models detect, discern, and assign the correct fine-grained description to objects. We further enhance our investigation by evaluating several state-of-the-art open-vocabulary object detectors using the proposed protocol.
  arXiv  Detail & Related papers  (2023-11-29T10:40:52Z)
• Toward Open Vocabulary Aerial Object Detection with CLIP-Activated Student-Teacher Learning [13.667326007851674]
  We propose CastDet, a CLIP-activated student-teacher open-vocabulary object detection framework. Our approach boosts not only novel object proposals but also classification. Experimental results demonstrate our CastDet achieving superior open-vocabulary detection performance.
  arXiv  Detail & Related papers  (2023-11-20T10:26:04Z)
• Open-Vocabulary Camouflaged Object Segmentation [66.94945066779988]
  We introduce a new task, open-vocabulary camouflaged object segmentation (OVCOS) We construct a large-scale complex scene dataset (textbfOVCamo) containing 11,483 hand-selected images with fine annotations and corresponding object classes. By integrating the guidance of class semantic knowledge and the supplement of visual structure cues from the edge and depth information, the proposed method can efficiently capture camouflaged objects.
  arXiv  Detail & Related papers  (2023-11-19T06:00:39Z)
• Described Object Detection: Liberating Object Detection with Flexible Expressions [19.392927971139652]
  We advance Open-Vocabulary object Detection (OVD) and Referring Expression (REC) to Described Object Detection (DOD) In this paper, we advance them to a more practical setting called Described Object Detection (DOD) by expanding category names to flexible language expressions for OVD. This dataset features flexible language expressions, whether short category names or long descriptions, and annotating all described objects on all images without omission.
  arXiv  Detail & Related papers  (2023-07-24T14:06:54Z)
• Learning Dense Object Descriptors from Multiple Views for Low-shot Category Generalization [27.583517870047487]
  We propose Deep Object Patch rimis (DOPE), which can be trained from multiple views of object instances without any category or semantic object part labels. To train DOPE, we assume access to sparse depths, foreground masks and known cameras, to obtain pixel-level correspondences between views of an object. We find that DOPE can directly be used for low-shot classification of novel categories using local-part matching, and is competitive with and outperforms supervised and self-supervised learning baselines.
  arXiv  Detail & Related papers  (2022-11-28T04:31:53Z)
• Learning Object-Language Alignments for Open-Vocabulary Object Detection [83.09560814244524]
  We propose a novel open-vocabulary object detection framework directly learning from image-text pair data. It enables us to train an open-vocabulary object detector on image-text pairs in a much simple and effective way.
  arXiv  Detail & Related papers  (2022-11-27T14:47:31Z)
• Incremental-DETR: Incremental Few-Shot Object Detection via Self-Supervised Learning [60.64535309016623]
  We propose the Incremental-DETR that does incremental few-shot object detection via fine-tuning and self-supervised learning on the DETR object detector. To alleviate severe over-fitting with few novel class data, we first fine-tune the class-specific components of DETR with self-supervision. We further introduce a incremental few-shot fine-tuning strategy with knowledge distillation on the class-specific components of DETR to encourage the network in detecting novel classes without catastrophic forgetting.
  arXiv  Detail & Related papers  (2022-05-09T05:08:08Z)
• Multi-Modal Few-Shot Object Detection with Meta-Learning-Based Cross-Modal Prompting [77.69172089359606]
  We study multi-modal few-shot object detection (FSOD) in this paper, using both few-shot visual examples and class semantic information for detection. Our approach is motivated by the high-level conceptual similarity of (metric-based) meta-learning and prompt-based learning. We comprehensively evaluate the proposed multi-modal FSOD models on multiple few-shot object detection benchmarks, achieving promising results.
  arXiv  Detail & Related papers  (2022-04-16T16:45:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.