CrossTune: Black-Box Few-Shot Classification with Label Enhancement

• URL: http://arxiv.org/abs/2403.12468v1
• Date: Tue, 19 Mar 2024 05:52:56 GMT
• Title: CrossTune: Black-Box Few-Shot Classification with Label Enhancement
• Authors: Danqing Luo, Chen Zhang, Yan Zhang, Haizhou Li,
• Abstract summary: We introduce a label-enhanced cross-attention network called CrossTune to study black-box language model adaptation without prompt search. Our proposed approach outperforms the previous state-of-the-art gradient-free black-box tuning method by 5.7% on average.
• Score: 40.88968135459357
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Training or finetuning large-scale language models (LLMs) requires substantial computation resources, motivating recent efforts to explore parameter-efficient adaptation to downstream tasks. One approach is to treat these models as black boxes and use forward passes (Inference APIs) to interact with them. Current research focuses on adapting these black-box models to downstream tasks using gradient-free prompt optimization, but this often involves an expensive process of searching task-specific prompts. Therefore, we are motivated to study black-box language model adaptation without prompt search. Specifically, we introduce a label-enhanced cross-attention network called CrossTune, which models the semantic relatedness between the input text sequence and task-specific label descriptions. Its effectiveness is examined in the context of few-shot text classification. To improve the generalization of CrossTune, we utilize ChatGPT to generate additional training data through in-context learning. A switch mechanism is implemented to exclude low-quality ChatGPT-generated data. Through extensive experiments on seven benchmark text classification datasets, we demonstrate that our proposed approach outperforms the previous state-of-the-art gradient-free black-box tuning method by 5.7% on average. Even without using ChatGPT-augmented data, CrossTune performs better or comparably than previous black-box tuning methods, suggesting the effectiveness of our approach.

Related papers

• Black-Box Forgetting [8.84485103053191]
  We address a novel problem of selective forgetting for black-box models, named Black-Box Forgetting. We propose Latent Context Sharing, which introduces common low-dimensional latent components among multiple tokens for the prompt. Experiments on four standard benchmark datasets demonstrate the superiority of our method with reasonable baselines.
  arXiv  Detail & Related papers  (2024-11-01T07:10:40Z)
• Adapting Vision-Language Models to Open Classes via Test-Time Prompt Tuning [50.26965628047682]
  Adapting pre-trained models to open classes is a challenging problem in machine learning. In this paper, we consider combining the advantages of both and come up with a test-time prompt tuning approach. Our proposed method outperforms all comparison methods on average considering both base and new classes.
  arXiv  Detail & Related papers  (2024-08-29T12:34:01Z)
• NC-TTT: A Noise Contrastive Approach for Test-Time Training [19.0284321951354]
  Noise-Contrastive Test-Time Training (NC-TTT) is a novel unsupervised TTT technique based on the discrimination of noisy feature maps. By learning to classify noisy views of projected feature maps, and then adapting the model accordingly on new domains, classification performance can be recovered by an important margin.
  arXiv  Detail & Related papers  (2024-04-12T10:54:11Z)
• Black-Box Tuning of Vision-Language Models with Effective Gradient Approximation [71.21346469382821]
  We introduce collaborative black-box tuning (CBBT) for both textual prompt optimization and output feature adaptation for black-box models. CBBT is extensively evaluated on eleven downstream benchmarks and achieves remarkable improvements compared to existing black-box VL adaptation methods.
  arXiv  Detail & Related papers  (2023-12-26T06:31:28Z)
• Enhancing Black-Box Few-Shot Text Classification with Prompt-Based Data Augmentation [42.05617728412819]
  We show how to optimize few-shot text classification without accessing the gradients of the large-scale language models. Our approach, dubbed BT-Classifier, significantly outperforms state-of-the-art black-box few-shot learners.
  arXiv  Detail & Related papers  (2023-05-23T07:54:34Z)
• Active Finetuning: Exploiting Annotation Budget in the Pretraining-Finetuning Paradigm [132.9949120482274]
  This paper focuses on the selection of samples for annotation in the pretraining-finetuning paradigm. We propose a novel method called ActiveFT for active finetuning task to select a subset of data distributing similarly with the entire unlabeled pool. Extensive experiments show the leading performance and high efficiency of ActiveFT superior to baselines on both image classification and semantic segmentation.
  arXiv  Detail & Related papers  (2023-03-25T07:17:03Z)
• Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models [107.05966685291067]
  We propose test-time prompt tuning (TPT) to learn adaptive prompts on the fly with a single test sample. TPT improves the zero-shot top-1 accuracy of CLIP by 3.6% on average. In evaluating cross-dataset generalization with unseen categories, TPT performs on par with the state-of-the-art approaches that use additional training data.
  arXiv  Detail & Related papers  (2022-09-15T17:55:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.