Unsupervised Pre-training with Language-Vision Prompts for Low-Data Instance Segmentation

• URL: http://arxiv.org/abs/2405.13388v1
• Date: Wed, 22 May 2024 06:48:43 GMT
• Title: Unsupervised Pre-training with Language-Vision Prompts for Low-Data Instance Segmentation
• Authors: Dingwen Zhang, Hao Li, Diqi He, Nian Liu, Lechao Cheng, Jingdong Wang, Junwei Han,
• Abstract summary: We propose a novel method for unsupervised pre-training in low-data regimes. Inspired by the recently successful prompting technique, we introduce a new method, Unsupervised Pre-training with Language-Vision Prompts. We show that our method can converge faster and perform better than CNN-based models in low-data regimes.
• Score: 105.23631749213729
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In recent times, following the paradigm of DETR (DEtection TRansformer), query-based end-to-end instance segmentation (QEIS) methods have exhibited superior performance compared to CNN-based models, particularly when trained on large-scale datasets. Nevertheless, the effectiveness of these QEIS methods diminishes significantly when confronted with limited training data. This limitation arises from their reliance on substantial data volumes to effectively train the pivotal queries/kernels that are essential for acquiring localization and shape priors. To address this problem, we propose a novel method for unsupervised pre-training in low-data regimes. Inspired by the recently successful prompting technique, we introduce a new method, Unsupervised Pre-training with Language-Vision Prompts (UPLVP), which improves QEIS models' instance segmentation by bringing language-vision prompts to queries/kernels. Our method consists of three parts: (1) Masks Proposal: Utilizes language-vision models to generate pseudo masks based on unlabeled images. (2) Prompt-Kernel Matching: Converts pseudo masks into prompts and injects the best-matched localization and shape features to their corresponding kernels. (3) Kernel Supervision: Formulates supervision for pre-training at the kernel level to ensure robust learning. With the help of our pre-training method, QEIS models can converge faster and perform better than CNN-based models in low-data regimes. Experimental evaluations conducted on MS COCO, Cityscapes, and CTW1500 datasets indicate that the QEIS models' performance can be significantly improved when pre-trained with our method. Code will be available at: https://github.com/lifuguan/UPLVP.

Related papers

• Data Adaptive Traceback for Vision-Language Foundation Models in Image Classification [34.37262622415682]
  We propose a new adaptation framework called Data Adaptive Traceback. Specifically, we utilize a zero-shot-based method to extract the most downstream task-related subset of the pre-training data. We adopt a pseudo-label-based semi-supervised technique to reuse the pre-training images and a vision-language contrastive learning method to address the confirmation bias issue in semi-supervised learning.
  arXiv  Detail & Related papers  (2024-07-11T18:01:58Z)
• Learning Semantic Proxies from Visual Prompts for Parameter-Efficient Fine-Tuning in Deep Metric Learning [13.964106147449051]
  Existing solutions concentrate on fine-tuning the pre-trained models on conventional image datasets. We propose a novel and effective framework based on learning Visual Prompts (VPT) in the pre-trained Vision Transformers (ViT) We demonstrate that our new approximations with semantic information are superior to representative capabilities.
  arXiv  Detail & Related papers  (2024-02-04T04:42:05Z)
• LAMM: Label Alignment for Multi-Modal Prompt Learning [17.478967970736115]
  We introduce an innovative label alignment method named textbfLAMM, which can adjust the category embeddings of downstream datasets through end-to-end training. Our method significantly improves the performance of existing multi-modal prompt learning models in few-shot scenarios. Our methodology exhibits the preeminence in continual learning compared to other prompt tuning methods.
  arXiv  Detail & Related papers  (2023-12-13T15:29:52Z)
• DST-Det: Simple Dynamic Self-Training for Open-Vocabulary Object Detection [72.25697820290502]
  This work introduces a straightforward and efficient strategy to identify potential novel classes through zero-shot classification. We refer to this approach as the self-training strategy, which enhances recall and accuracy for novel classes without requiring extra annotations, datasets, and re-training. Empirical evaluations on three datasets, including LVIS, V3Det, and COCO, demonstrate significant improvements over the baseline performance.
  arXiv  Detail & Related papers  (2023-10-02T17:52:24Z)
• Boosting Low-Data Instance Segmentation by Unsupervised Pre-training with Saliency Prompt [103.58323875748427]
  This work offers a novel unsupervised pre-training solution for low-data regimes. Inspired by the recent success of the Prompting technique, we introduce a new pre-training method that boosts QEIS models. Experimental results show that our method significantly boosts several QEIS models on three datasets.
  arXiv  Detail & Related papers  (2023-02-02T15:49:03Z)
• Improving Pre-trained Language Model Fine-tuning with Noise Stability Regularization [94.4409074435894]
  We propose a novel and effective fine-tuning framework, named Layerwise Noise Stability Regularization (LNSR) Specifically, we propose to inject the standard Gaussian noise and regularize hidden representations of the fine-tuned model. We demonstrate the advantages of the proposed method over other state-of-the-art algorithms including L2-SP, Mixout and SMART.
  arXiv  Detail & Related papers  (2022-06-12T04:42:49Z)
• SimVLM: Simple Visual Language Model Pretraining with Weak Supervision [48.98275876458666]
  We present a minimalist pretraining framework, named Simple Visual Language Model (SimVLM) SimVLM reduces the training complexity by exploiting large-scale weak supervision. It achieves new state-of-the-art results on a wide range of discriminative and generative vision-language benchmarks.
  arXiv  Detail & Related papers  (2021-08-24T18:14:00Z)
• Prior Guided Feature Enrichment Network for Few-Shot Segmentation [64.91560451900125]
  State-of-the-art semantic segmentation methods require sufficient labeled data to achieve good results. Few-shot segmentation is proposed to tackle this problem by learning a model that quickly adapts to new classes with a few labeled support samples. Theses frameworks still face the challenge of generalization ability reduction on unseen classes due to inappropriate use of high-level semantic information.
  arXiv  Detail & Related papers  (2020-08-04T10:41:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.