Related papers: Improving and Simplifying Pattern Exploiting Training

Improving and Simplifying Pattern Exploiting Training

URL: http://arxiv.org/abs/2103.11955v1
Date: Mon, 22 Mar 2021 15:52:45 GMT
Title: Improving and Simplifying Pattern Exploiting Training
Authors: Derek Tam, Rakesh R Menon, Mohit Bansal, Shashank Srivastava, Colin Raffel
Abstract summary: Pattern Exploiting Training (PET) is a recent approach that leverages patterns for few-shot learning. In this paper, we focus on few shot learning without any unlabeled data and introduce ADAPET. ADAPET outperforms PET on SuperGLUE without any task-specific unlabeled data.
Score: 81.77863825517511
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Abstract: Recently, pre-trained language models (LMs) have achieved strong performance when fine-tuned on difficult benchmarks like SuperGLUE. However, performance can suffer when there are very few labeled examples available for fine-tuning. Pattern Exploiting Training (PET) is a recent approach that leverages patterns for few-shot learning. However, PET uses task-specific unlabeled data. In this paper, we focus on few shot learning without any unlabeled data and introduce ADAPET, which modifies PET's objective to provide denser supervision during fine-tuning. As a result, ADAPET outperforms PET on SuperGLUE without any task-specific unlabeled data. Our code can be found at https://github.com/rrmenon10/ADAPET.

Related papers

Data Diet: Can Trimming PET/CT Datasets Enhance Lesion Segmentation? [68.51625420337575]
We describe our approach to compete in the autoPET3 datacentric track. We find that in the autoPETIII dataset, a model that is trained on the entire dataset exhibits undesirable characteristics. We counteract this by removing the easiest samples from the training dataset as measured by the model loss before retraining from scratch.
arXiv Detail & Related papers (2024-09-20T14:47:58Z)
Uncertainty-aware Parameter-Efficient Self-training for Semi-supervised Language Understanding [38.11411155621616]
We study self-training as one of the predominant semi-supervised learning approaches. We present UPET, a novel Uncertainty-aware self-Training framework. We show that UPET achieves a substantial improvement in terms of performance and efficiency.
arXiv Detail & Related papers (2023-10-19T02:18:29Z)
ConPET: Continual Parameter-Efficient Tuning for Large Language Models [65.48107393731861]
Continual learning requires continual adaptation of models to newly emerging tasks. We propose Continual. Efficient Tuning (ConPET), a generalizable paradigm for. continual task adaptation of large language models.
arXiv Detail & Related papers (2023-09-26T08:52:04Z)
Few-shot learning for sentence pair classification and its applications in software engineering [0.36832029288386137]
This work is to investigate the performance of alternative few-shot learning approaches with BERT-based models. vanilla fine-tuning, PET and SetFit are compared for numerous BERT-based checkpoints over an array of training set sizes. Our results establish PET as a strong few-shot learning approach, and our analysis shows that with just a few hundred labeled examples it can achieve performance near that of fine-tuning on full-sized data sets.
arXiv Detail & Related papers (2023-06-13T18:23:52Z)
Exploring the Impact of Model Scaling on Parameter-Efficient Tuning [100.61202305296275]
Scaling-efficient tuning (PET) methods can effectively drive extremely large pre-trained language models (PLMs) In small PLMs, there are usually noticeable performance differences among PET methods. We introduce a more flexible PET method called Arbitrary PET (APET) method.
arXiv Detail & Related papers (2023-06-04T10:10:54Z)
A Unified Continual Learning Framework with General Parameter-Efficient Tuning [56.250772378174446]
"Pre-training $rightarrow$ downstream adaptation" presents both new opportunities and challenges for Continual Learning. We position prompting as one instantiation of PET, and propose a unified CL framework, dubbed as Learning-Accumulation-Ensemble (LAE) PET, e.g., using Adapter, LoRA, or Prefix, can adapt a pre-trained model to downstream tasks with fewer parameters and resources.
arXiv Detail & Related papers (2023-03-17T15:52:45Z)
Active PETs: Active Data Annotation Prioritisation for Few-Shot Claim Verification with Pattern Exploiting Training [21.842139093124512]
Active PETs is a weighted approach that actively selects unlabelled data as candidates for annotation. Using Active PETs for data selection shows consistent improvement over the state-of-the-art active learning method. Our approach enables effective selection of instances to be labelled where unlabelled data is abundant.
arXiv Detail & Related papers (2022-08-18T10:11:36Z)
Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning [81.3514358542452]
Few-shot in-context learning (ICL) incurs substantial computational, memory, and storage costs because it involves processing all of the training examples every time a prediction is made. parameter-efficient fine-tuning offers an alternative paradigm where a small set of parameters are trained to enable a model to perform the new task. In this paper, we rigorously compare few-shot ICL and parameter-efficient fine-tuning and demonstrate that the latter offers better accuracy as well as dramatically lower computational costs.
arXiv Detail & Related papers (2022-05-11T17:10:41Z)
True Few-Shot Learning with Prompts -- A Real-World Perspective [12.919486518128734]
PET is a method that combines textual instructions with example-based finetuning. We show that, if correctly configured, PET performs strongly in a true few-shot setting, without a dev set. We then put our findings to a real-world test by running PET on RAFT, a benchmark of tasks taken directly from realistic NLP applications.
arXiv Detail & Related papers (2021-11-26T11:49:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.