CASA: Class-Agnostic Shared Attributes in Vision-Language Models for Efficient Incremental Object Detection

• URL: http://arxiv.org/abs/2410.05804v2
• Date: Fri, 11 Oct 2024 08:54:41 GMT
• Title: CASA: Class-Agnostic Shared Attributes in Vision-Language Models for Efficient Incremental Object Detection
• Authors: Mingyi Guo, Yuyang Liu, Zongying Lin, Peixi Peng, Yonghong Tian,
• Abstract summary: We propose a novel method utilizing attributes in vision-language foundation models for incremental object detection. Our method constructs a Class-Agnostic Shared Attribute base (CASA) to capture common semantic information among incremental classes. Our method adds only 0.7% to parameter storage through parameter-efficient fine-tuning to significantly enhance the scalability and adaptability of our proposed method.
• Score: 30.46562066023117
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Incremental object detection (IOD) is challenged by background shift, where background categories in sequential data may include previously learned or future classes. Inspired by the vision-language foundation models such as CLIP, these models capture shared attributes from extensive image-text paired data during pre-training. We propose a novel method utilizing attributes in vision-language foundation models for incremental object detection. Our method constructs a Class-Agnostic Shared Attribute base (CASA) to capture common semantic information among incremental classes. Specifically, we utilize large language models to generate candidate textual attributes and select the most relevant ones based on current training data, recording their significance in an attribute assignment matrix. For subsequent tasks, we freeze the retained attributes and continue selecting from the remaining candidates while updating the attribute assignment matrix accordingly. Furthermore, we employ OWL-ViT as our baseline, preserving the original parameters of the pre-trained foundation model. Our method adds only 0.7% to parameter storage through parameter-efficient fine-tuning to significantly enhance the scalability and adaptability of IOD. Extensive two-phase and multi-phase experiments on the COCO dataset demonstrate the state-of-the-art performance of our proposed method.

Related papers

• Spatio-Temporal Side Tuning Pre-trained Foundation Models for Video-based Pedestrian Attribute Recognition [58.79807861739438]
  Existing pedestrian recognition (PAR) algorithms are mainly developed based on a static image. We propose to understand human attributes using video frames that can fully use temporal information.
  arXiv  Detail & Related papers  (2024-04-27T14:43:32Z)
• Exploiting Contextual Target Attributes for Target Sentiment Classification [53.30511968323911]
  Existing PTLM-based models for TSC can be categorized into two groups: 1) fine-tuning-based models that adopt PTLM as the context encoder; 2) prompting-based models that transfer the classification task to the text/word generation task. We present a new perspective of leveraging PTLM for TSC: simultaneously leveraging the merits of both language modeling and explicit target-context interactions via contextual target attributes.
  arXiv  Detail & Related papers  (2023-12-21T11:45:28Z)
• UP-DP: Unsupervised Prompt Learning for Data Pre-Selection with Vision-Language Models [24.50445616970387]
  We introduce UP-DP, a simple yet effective unsupervised prompt learning approach that adapts vision-language models for data pre-selection. Specifically, with the BLIP-2 parameters frozen, we train text prompts to extract the joint features with improved representation. We extensively compare our method with the state-of-the-art using seven benchmark datasets in different settings, achieving up to a performance gain of 20%.
  arXiv  Detail & Related papers  (2023-07-20T20:45:13Z)
• Leveraging Vision-Language Foundation Models for Fine-Grained Downstream Tasks [17.367599062853156]
  Vision-language foundation models such as CLIP have shown impressive zero-shot performance on many tasks and datasets. We propose a multitask fine-tuning strategy based on a positive/negative prompt formulation to further leverage the capacities of the vision-language foundation models.
  arXiv  Detail & Related papers  (2023-07-13T15:05:34Z)
• Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Bias [92.41919689753051]
  Large language models (LLMs) have been recently leveraged as training data generators for various natural language processing (NLP) tasks. We investigate training data generation with diversely attributed prompts, which have the potential to yield diverse and attributed generated data. We show that attributed prompts outperform simple class-conditional prompts in terms of the resulting model's performance.
  arXiv  Detail & Related papers  (2023-06-28T03:31:31Z)
• OvarNet: Towards Open-vocabulary Object Attribute Recognition [42.90477523238336]
  We propose a naive two-stage approach for open-vocabulary object detection and attribute classification, termed CLIP-Attr. The candidate objects are first proposed with an offline RPN and later classified for semantic category and attributes. We show that recognition of semantic category and attributes is complementary for visual scene understanding.
  arXiv  Detail & Related papers  (2023-01-23T15:59:29Z)
• Improving Meta-learning for Low-resource Text Classification and Generation via Memory Imitation [87.98063273826702]
  We propose a memory imitation meta-learning (MemIML) method that enhances the model's reliance on support sets for task adaptation. A theoretical analysis is provided to prove the effectiveness of our method.
  arXiv  Detail & Related papers  (2022-03-22T12:41:55Z)
• Efficient Attribute Injection for Pretrained Language Models [20.39972635495006]
  We propose a lightweight and memory-efficient method to inject attributes to pretrained language models (PLMs) To limit the increase of parameters especially when the attribute vocabulary is large, we use low-rank approximations and hypercomplex multiplications. Our method outperforms previous attribute injection methods and achieves state-of-the-art performance on various datasets.
  arXiv  Detail & Related papers  (2021-09-16T13:08:24Z)
• Selecting Relevant Features from a Multi-domain Representation for Few-shot Classification [91.67977602992657]
  We propose a new strategy based on feature selection, which is both simpler and more effective than previous feature adaptation approaches. We show that a simple non-parametric classifier built on top of such features produces high accuracy and generalizes to domains never seen during training.
  arXiv  Detail & Related papers  (2020-03-20T15:44:17Z)
• Pairwise Similarity Knowledge Transfer for Weakly Supervised Object Localization [53.99850033746663]
  We study the problem of learning localization model on target classes with weakly supervised image labels. In this work, we argue that learning only an objectness function is a weak form of knowledge transfer. Experiments on the COCO and ILSVRC 2013 detection datasets show that the performance of the localization model improves significantly with the inclusion of pairwise similarity function.
  arXiv  Detail & Related papers  (2020-03-18T17:53:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.