PromptMix: A Class Boundary Augmentation Method for Large Language Model
Distillation
- URL: http://arxiv.org/abs/2310.14192v1
- Date: Sun, 22 Oct 2023 05:43:23 GMT
- Title: PromptMix: A Class Boundary Augmentation Method for Large Language Model
Distillation
- Authors: Gaurav Sahu, Olga Vechtomova, Dzmitry Bahdanau, Issam H. Laradji
- Abstract summary: We propose a method to generate more helpful augmented data by utilizing the LLM's abilities to follow instructions and perform few-shot classifications.
Our specific PromptMix method consists of two steps: 1) generate challenging text augmentations near class boundaries; however, generating borderline examples increases the risk of false positives in the dataset.
We evaluate the proposed method in challenging 2-shot and zero-shot settings on four text classification datasets: Banking77, TREC6, Subjectivity (SUBJ) and Twitter Complaints.
- Score: 19.351192775314612
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Data augmentation is a widely used technique to address the problem of text
classification when there is a limited amount of training data. Recent work
often tackles this problem using large language models (LLMs) like GPT3 that
can generate new examples given already available ones. In this work, we
propose a method to generate more helpful augmented data by utilizing the LLM's
abilities to follow instructions and perform few-shot classifications. Our
specific PromptMix method consists of two steps: 1) generate challenging text
augmentations near class boundaries; however, generating borderline examples
increases the risk of false positives in the dataset, so we 2) relabel the text
augmentations using a prompting-based LLM classifier to enhance the correctness
of labels in the generated data. We evaluate the proposed method in challenging
2-shot and zero-shot settings on four text classification datasets: Banking77,
TREC6, Subjectivity (SUBJ), and Twitter Complaints. Our experiments show that
generating and, crucially, relabeling borderline examples facilitates the
transfer of knowledge of a massive LLM like GPT3.5-turbo into smaller and
cheaper classifiers like DistilBERT$_{base}$ and BERT$_{base}$. Furthermore,
2-shot PromptMix outperforms multiple 5-shot data augmentation methods on the
four datasets. Our code is available at
https://github.com/ServiceNow/PromptMix-EMNLP-2023.
Related papers
- An Automatic Prompt Generation System for Tabular Data Tasks [3.117741687220381]
Large language models (LLMs) have demonstrated their ability on several tasks through carefully crafted prompts.
This paper presents an innovative auto-prompt generation system suitable for multiple LLMs, with minimal training.
arXiv Detail & Related papers (2024-05-09T08:32:55Z) - Leveraging Weakly Annotated Data for Hate Speech Detection in Code-Mixed
Hinglish: A Feasibility-Driven Transfer Learning Approach with Large Language
Models [0.0]
Hate speech detection in mix-code low-resource languages is an active problem area where the use of Large Language Models has proven beneficial.
In this study, we have compiled a dataset of 100 YouTube comments, and weakly labelled them for coarse and fine-grained misogyny classification in mix-code Hinglish.
Out of all the approaches, zero-shot classification using the Bidirectional Auto-Regressive Transformers (BART) large model and few-shot prompting using Generative Pre-trained Transformer- 3 (ChatGPT-3) achieve the best results.
arXiv Detail & Related papers (2024-03-04T15:27:49Z) - Learning to Prompt with Text Only Supervision for Vision-Language Models [107.282881515667]
One branch of methods adapts CLIP by learning prompts using visual information.
An alternative approach resorts to training-free methods by generating class descriptions from large language models.
We propose to combine the strengths of both streams by learning prompts using only text data.
arXiv Detail & Related papers (2024-01-04T18:59:49Z) - AnnoLLM: Making Large Language Models to Be Better Crowdsourced Annotators [98.11286353828525]
GPT-3.5 series models have demonstrated remarkable few-shot and zero-shot ability across various NLP tasks.
We propose AnnoLLM, which adopts a two-step approach, explain-then-annotate.
We build the first conversation-based information retrieval dataset employing AnnoLLM.
arXiv Detail & Related papers (2023-03-29T17:03:21Z) - Distinguishability Calibration to In-Context Learning [31.375797763897104]
We propose a method to map a PLM-encoded embedding into a new metric space to guarantee the distinguishability of the resulting embeddings.
We also take the advantage of hyperbolic embeddings to capture the hierarchical relations among fine-grained class-associated token embedding.
arXiv Detail & Related papers (2023-02-13T09:15:00Z) - An Embarrassingly Simple Approach to Semi-Supervised Few-Shot Learning [58.59343434538218]
We propose a simple but quite effective approach to predict accurate negative pseudo-labels of unlabeled data from an indirect learning perspective.
Our approach can be implemented in just few lines of code by only using off-the-shelf operations.
arXiv Detail & Related papers (2022-09-28T02:11:34Z) - DoubleMix: Simple Interpolation-Based Data Augmentation for Text
Classification [56.817386699291305]
This paper proposes a simple yet effective data augmentation approach termed DoubleMix.
DoubleMix first generates several perturbed samples for each training data.
It then uses the perturbed data and original data to carry out a two-step in the hidden space of neural models.
arXiv Detail & Related papers (2022-09-12T15:01:04Z) - Revisiting Self-Training for Few-Shot Learning of Language Model [61.173976954360334]
Unlabeled data carry rich task-relevant information, they are proven useful for few-shot learning of language model.
In this work, we revisit the self-training technique for language model fine-tuning and present a state-of-the-art prompt-based few-shot learner, SFLM.
arXiv Detail & Related papers (2021-10-04T08:51:36Z) - MixText: Linguistically-Informed Interpolation of Hidden Space for
Semi-Supervised Text Classification [68.15015032551214]
MixText is a semi-supervised learning method for text classification.
TMix creates a large amount of augmented training samples by interpolating text in hidden space.
We leverage recent advances in data augmentation to guess low-entropy labels for unlabeled data.
arXiv Detail & Related papers (2020-04-25T21:37:36Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.