SelfPrompt: Confidence-Aware Semi-Supervised Tuning for Robust Vision-Language Model Adaptation

• URL: http://arxiv.org/abs/2501.14148v2
• Date: Tue, 28 Jan 2025 23:21:56 GMT
• Title: SelfPrompt: Confidence-Aware Semi-Supervised Tuning for Robust Vision-Language Model Adaptation
• Authors: Shuvendu Roy, Ali Etemad,
• Abstract summary: SelfPrompt is a novel prompt-tuning approach for vision-language models (VLMs) in a semi-supervised learning setup.<n>We introduce a cluster-guided pseudo-labelling method that improves pseudo-label accuracy.<n>We also present a confidence-aware semi-supervised learning module that maximizes the utilization of unlabelled data.
• Score: 23.4909421082857
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present SelfPrompt, a novel prompt-tuning approach for vision-language models (VLMs) in a semi-supervised learning setup. Existing methods for tuning VLMs in semi-supervised setups struggle with the negative impact of the miscalibrated VLMs on pseudo-labelling, and the accumulation of noisy pseudo-labels. SelfPrompt addresses these challenges by introducing a cluster-guided pseudo-labelling method that improves pseudo-label accuracy, and a confidence-aware semi-supervised learning module that maximizes the utilization of unlabelled data by combining supervised learning and weakly-supervised learning. Additionally, we investigate our method in an active semi-supervised learning setup, where the labelled set is strategically selected to ensure the best utilization of a limited labelling budget. To this end, we propose a weakly-supervised sampling technique that selects a diverse and representative labelled set, which can be seamlessly integrated into existing methods to enhance their performance. We conduct extensive evaluations across 13 datasets, significantly surpassing state-of-the-art performances with average improvements of 6.23% in standard semi-supervised learning, 6.25% in active semi-supervised learning, and 4.9% in base-to-novel generalization, using a 2-shot setup. Furthermore, SelfPrompt shows excellent generalization in single-shot settings, achieving an average improvement of 11.78%.

Related papers

• Feedback-Driven Pseudo-Label Reliability Assessment: Redefining Thresholding for Semi-Supervised Semantic Segmentation [5.7977777220041204]
  A common practice in pseudo-supervision is filtering pseudo-labels based on pre-defined confidence thresholds or entropy.<n>We propose Ensemble-of-Confidence Reinforcement (ENCORE), a dynamic feedback-driven thresholding strategy for pseudo-label selection.<n>Our method seamlessly integrates into existing pseudo-supervision frameworks and significantly improves segmentation performance.
  arXiv  Detail & Related papers  (2025-05-12T15:58:08Z)
• Maximally Separated Active Learning [32.98415531556376]
  We propose an active learning method that utilizes fixed equiangular hyperspherical points as class prototypes. We demonstrate strong performance over existing active learning techniques across five benchmark datasets.
  arXiv  Detail & Related papers  (2024-11-26T14:02:43Z)
• Dual-Decoupling Learning and Metric-Adaptive Thresholding for Semi-Supervised Multi-Label Learning [81.83013974171364]
  Semi-supervised multi-label learning (SSMLL) is a powerful framework for leveraging unlabeled data to reduce the expensive cost of collecting precise multi-label annotations.<n>Unlike semi-supervised learning, one cannot select the most probable label as the pseudo-label in SSMLL due to multiple semantics contained in an instance.<n>We propose a dual-perspective method to generate high-quality pseudo-labels.
  arXiv  Detail & Related papers  (2024-07-26T09:33:53Z)
• Unsupervised End-to-End Training with a Self-Defined Target [2.6563873893593826]
  We propose a method enabling networks or hardware designed for end-to-end supervised learning to also perform high-performance unsupervised learning. We extend this method to semi-supervised learning, adjusting targets based on data type, achieving 96.6% accuracy with only 600 labeled MNIST samples in a multi-layer perceptron.
  arXiv  Detail & Related papers  (2024-03-18T16:14:28Z)
• BAL: Balancing Diversity and Novelty for Active Learning [53.289700543331925]
  We introduce a novel framework, Balancing Active Learning (BAL), which constructs adaptive sub-pools to balance diverse and uncertain data. Our approach outperforms all established active learning methods on widely recognized benchmarks by 1.20%.
  arXiv  Detail & Related papers  (2023-12-26T08:14:46Z)
• Semi-Supervised Class-Agnostic Motion Prediction with Pseudo Label Regeneration and BEVMix [59.55173022987071]
  We study the potential of semi-supervised learning for class-agnostic motion prediction. Our framework adopts a consistency-based self-training paradigm, enabling the model to learn from unlabeled data. Our method exhibits comparable performance to weakly and some fully supervised methods.
  arXiv  Detail & Related papers  (2023-12-13T09:32:50Z)
• A Unified Approach to Semi-Supervised Out-of-Distribution Detection [0.0]
  One of the early weaknesses identified in. deep training was their inability to provide low confidence predictions. MoLAR is efficient to train, and provides strong. OOD performance when only comparing the exemplars to a small set of images chosen to be representative.
  arXiv  Detail & Related papers  (2023-11-28T06:12:28Z)
• One-bit Supervision for Image Classification: Problem, Solution, and Beyond [114.95815360508395]
  This paper presents one-bit supervision, a novel setting of learning with fewer labels, for image classification. We propose a multi-stage training paradigm and incorporate negative label suppression into an off-the-shelf semi-supervised learning algorithm. In multiple benchmarks, the learning efficiency of the proposed approach surpasses that using full-bit, semi-supervised supervision.
  arXiv  Detail & Related papers  (2023-11-26T07:39:00Z)
• Scaling Up Semi-supervised Learning with Unconstrained Unlabelled Data [27.75143621836449]
  We propose UnMixMatch, a semi-supervised learning framework which can learn effective representations from unconstrained data. We perform extensive experiments on 4 commonly used datasets and demonstrate superior performance over existing semi-supervised methods with a performance boost of 4.79%.
  arXiv  Detail & Related papers  (2023-06-02T01:07:14Z)
• Class-Aware Contrastive Semi-Supervised Learning [51.205844705156046]
  We propose a general method named Class-aware Contrastive Semi-Supervised Learning (CCSSL) to improve pseudo-label quality and enhance the model's robustness in the real-world setting. Our proposed CCSSL has significant performance improvements over the state-of-the-art SSL methods on the standard datasets CIFAR100 and STL10.
  arXiv  Detail & Related papers  (2022-03-04T12:18:23Z)
• Improving Robustness and Efficiency in Active Learning with Contrastive Loss [13.994967246046008]
  This paper introduces supervised contrastive active learning (SCAL) by leveraging the contrastive loss for active learning in a supervised setting. We propose efficient query strategies in active learning to select unbiased and informative data samples of diverse feature representations.
  arXiv  Detail & Related papers  (2021-09-13T21:09:21Z)
• Semi-supervised Batch Active Learning via Bilevel Optimization [89.37476066973336]
  We formulate our approach as a data summarization problem via bilevel optimization. We show that our method is highly effective in keyword detection tasks in the regime when only few labeled samples are available.
  arXiv  Detail & Related papers  (2020-10-19T16:53:24Z)
• A Comprehensive Approach to Unsupervised Embedding Learning based on AND Algorithm [18.670975246545208]
  Unsupervised embedding learning aims to extract good representation from data without the need for any manual labels. This paper proposes a new unsupervised embedding approach, called Super-AND, which extends the current state-of-the-art model. Super-AND outperforms all existing approaches and achieves an accuracy of 89.2% on the image classification task for CIFAR-10.
  arXiv  Detail & Related papers  (2020-02-26T13:22:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.