VLM-PL: Advanced Pseudo Labeling Approach for Class Incremental Object Detection via Vision-Language Model

• URL: http://arxiv.org/abs/2403.05346v3
• Date: Thu, 9 May 2024 00:56:29 GMT
• Title: VLM-PL: Advanced Pseudo Labeling Approach for Class Incremental Object Detection via Vision-Language Model
• Authors: Junsu Kim, Yunhoe Ku, Jihyeon Kim, Junuk Cha, Seungryul Baek,
• Abstract summary: We introduce Vision-Language Model assisted Pseudo-Labeling (VLM-PL) This technique uses Vision-Language Model (VLM) to verify the correctness of pseudo ground-truths (GTs) without requiring additional model training. VLM-PL integrates refined pseudo and real GTs from upcoming training, effectively combining new and old knowledge.
• Score: 9.122593534510512
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In the field of Class Incremental Object Detection (CIOD), creating models that can continuously learn like humans is a major challenge. Pseudo-labeling methods, although initially powerful, struggle with multi-scenario incremental learning due to their tendency to forget past knowledge. To overcome this, we introduce a new approach called Vision-Language Model assisted Pseudo-Labeling (VLM-PL). This technique uses Vision-Language Model (VLM) to verify the correctness of pseudo ground-truths (GTs) without requiring additional model training. VLM-PL starts by deriving pseudo GTs from a pre-trained detector. Then, we generate custom queries for each pseudo GT using carefully designed prompt templates that combine image and text features. This allows the VLM to classify the correctness through its responses. Furthermore, VLM-PL integrates refined pseudo and real GTs from upcoming training, effectively combining new and old knowledge. Extensive experiments conducted on the Pascal VOC and MS COCO datasets not only highlight VLM-PL's exceptional performance in multi-scenario but also illuminate its effectiveness in dual-scenario by achieving state-of-the-art results in both.

Related papers

• MarvelOVD: Marrying Object Recognition and Vision-Language Models for Robust Open-Vocabulary Object Detection [107.15164718585666]
  We investigate the root cause of VLMs' biased prediction under the open vocabulary detection context. Our observations lead to a simple yet effective paradigm, coded MarvelOVD, that generates significantly better training targets. Our method outperforms the other state-of-the-arts by significant margins.
  arXiv  Detail & Related papers  (2024-07-31T09:23:57Z)
• VILA: On Pre-training for Visual Language Models [74.08039416548209]
  We study the design options for VLM pre-training through step-by-step controllable comparisons. We build VILA, a Visual Language model family that consistently outperforms the state-of-the-art models.
  arXiv  Detail & Related papers  (2023-12-12T18:58:18Z)
• LION : Empowering Multimodal Large Language Model with Dual-Level Visual Knowledge [58.82222646803248]
  Multimodal Large Language Models (MLLMs) have endowed LLMs with the ability to perceive and understand multi-modal signals. Most of the existing MLLMs mainly adopt vision encoders pretrained on coarsely aligned image-text pairs, leading to insufficient extraction and reasoning of visual knowledge. We propose a dual-Level vIsual knedgeOwl eNhanced Multimodal Large Language Model (LION), which empowers the MLLM by injecting visual knowledge in two levels.
  arXiv  Detail & Related papers  (2023-11-20T15:56:44Z)
• Active Prompt Learning in Vision Language Models [21.276006224504748]
  We devise a novel active learning framework for pre-trained Vision Language Models, denoted as PCB. We conduct experiments on seven different real-world datasets, and the results demonstrate that PCB surpasses conventional active learning and random sampling methods.
  arXiv  Detail & Related papers  (2023-11-18T22:42:16Z)
• Adapting Pre-trained Language Models to Vision-Language Tasks via Dynamic Visual Prompting [83.21164539349273]
  Pre-trained language models (PLMs) have played an increasing role in multimedia research. In this paper, we focus on exploring PLMs as a stand-alone model for vision-language reasoning tasks. We propose a novel transfer learning approach for PLMs, termed Dynamic Visual Prompting (DVP)
  arXiv  Detail & Related papers  (2023-06-01T07:19:28Z)
• Learning without Forgetting for Vision-Language Models [65.49600786387106]
  Class-Incremental Learning (CIL) or continual learning is a desired capability in the real world. Recent advances in Vision-Language Models (VLM) have shown promising capabilities in learning generalizable representations. We propose PROjectiOn Fusion (PROOF) that enables VLMs to learn without forgetting.
  arXiv  Detail & Related papers  (2023-05-30T17:59:32Z)
• Fine-grained Visual-Text Prompt-Driven Self-Training for Open-Vocabulary Object Detection [87.39089806069707]
  We propose a fine-grained Visual-Text Prompt-driven self-training paradigm for Open-Vocabulary Detection (VTP-OVD) During the adapting stage, we enable VLM to obtain fine-grained alignment by using learnable text prompts to resolve an auxiliary dense pixel-wise prediction task. Experiments show that our method achieves the state-of-the-art performance for open-vocabulary object detection, e.g., 31.5% mAP on unseen classes of COCO.
  arXiv  Detail & Related papers  (2022-11-02T03:38:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.