Improving Short Text Classification With Augmented Data Using GPT-3

• URL: http://arxiv.org/abs/2205.10981v1
• Date: Mon, 23 May 2022 01:10:38 GMT
• Title: Improving Short Text Classification With Augmented Data Using GPT-3
• Authors: Salvador Balkus and Donghui Yan
• Abstract summary: GPT-3 is a large-scale natural language model developed by OpenAI. This study teaches GPT-3 to classify whether a question is related to data science by augmenting a small training set with additional examples. We find that while the augmented Completion achieves upwards of 80 percent validation accuracy, using the augmented Classification yields more consistent accuracy on unseen examples.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: GPT-3 is a large-scale natural language model developed by OpenAI that can perform many different tasks, including topic classification. Although researchers claim that it requires only a small number of in-context examples to learn a task, in practice GPT-3 requires these training examples to be either of exceptional quality or a higher quantity than easily created by hand. To address this issue, this study teaches GPT-3 to classify whether a question is related to data science by augmenting a small training set with additional examples generated by GPT-3 itself. This study compares two classifiers: the GPT-3 Classification Endpoint with augmented examples, and the GPT-3 Completion Endpoint with an optimal training set chosen using a genetic algorithm. We find that while the augmented Completion Endpoint achieves upwards of 80 percent validation accuracy, using the augmented Classification Endpoint yields more consistent accuracy on unseen examples. In this way, giving large-scale machine learning models like GPT-3 the ability to propose their own additional training examples can result in improved classification performance.

Related papers

• Text generation for dataset augmentation in security classification tasks [55.70844429868403]
  This study evaluates the application of natural language text generators to fill this data gap in multiple security-related text classification tasks. We find substantial benefits for GPT-3 data augmentation strategies in situations with severe limitations on known positive-class samples.
  arXiv  Detail & Related papers  (2023-10-22T22:25:14Z)
• SentimentGPT: Exploiting GPT for Advanced Sentiment Analysis and its Departure from Current Machine Learning [5.177947445379688]
  This study presents a thorough examination of various Generative Pretrained Transformer (GPT) methodologies in sentiment analysis. Three primary strategies are employed: 1) prompt engineering using the advanced GPT-3.5 Turbo, 2) fine-tuning GPT models, and 3) an inventive approach to embedding classification. The research yields detailed comparative insights among these strategies and individual GPT models, revealing their unique strengths and potential limitations.
  arXiv  Detail & Related papers  (2023-07-16T05:33:35Z)
• Is GPT-3 a Good Data Annotator? [30.9559541574174]
  GPT-3 is a large-scale language model developed by OpenAI. In this paper, we evaluate the performance of GPT-3 as a data annotator.
  arXiv  Detail & Related papers  (2022-12-20T17:28:41Z)
• Thinking about GPT-3 In-Context Learning for Biomedical IE? Think Again [24.150464908060112]
  We present the first systematic and comprehensive study to compare the few-shot performance of GPT-3 in-context learning with fine-tuning smaller (i.e., BERT-sized) PLMs. Our results show that GPT-3 still significantly underperforms compared with simply fine-tuning a smaller PLM using the same small training set.
  arXiv  Detail & Related papers  (2022-03-16T05:56:08Z)
• Guiding Generative Language Models for Data Augmentation in Few-Shot Text Classification [59.698811329287174]
  We leverage GPT-2 for generating artificial training instances in order to improve classification performance. Our results show that fine-tuning GPT-2 in a handful of label instances leads to consistent classification improvements.
  arXiv  Detail & Related papers  (2021-11-17T12:10:03Z)
• Reframing Instructional Prompts to GPTk's Language [72.69833640335519]
  We propose reframing techniques for model designers to create effective prompts for language models. Our results show that reframing improves few-shot learning performance by 14% while reducing sample complexity. The performance gains are particularly important on large language models, such as GPT3 where tuning models or prompts on large datasets is not feasible.
  arXiv  Detail & Related papers  (2021-09-16T09:44:43Z)
• Calibrate Before Use: Improving Few-Shot Performance of Language Models [68.17016463756474]
  GPT-3 can perform numerous tasks when provided a natural language prompt that contains a few training examples. We show that this type of few-shot learning can be unstable. The choice of prompt format, training examples, and even the order of the training examples can cause accuracy to vary from near chance to near state-of-the-art.
  arXiv  Detail & Related papers  (2021-02-19T00:23:59Z)
• What Makes Good In-Context Examples for GPT-$3$? [101.99751777056314]
  GPT-$3$ has attracted lots of attention due to its superior performance across a wide range of NLP tasks. Despite its success, we found that the empirical results of GPT-$3$ depend heavily on the choice of in-context examples. In this work, we investigate whether there are more effective strategies for judiciously selecting in-context examples.
  arXiv  Detail & Related papers  (2021-01-17T23:38:40Z)
• Language Models are Few-Shot Learners [61.36677350504291]
  We show that scaling up language models greatly improves task-agnostic, few-shot performance. We train GPT-3, an autoregressive language model with 175 billion parameters, and test its performance in the few-shot setting. GPT-3 achieves strong performance on many NLP datasets, including translation, question-answering, and cloze tasks.
  arXiv  Detail & Related papers  (2020-05-28T17:29:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.