LLMs vs Established Text Augmentation Techniques for Classification: When do the Benefits Outweight the Costs?

• URL: http://arxiv.org/abs/2408.16502v1
• Date: Thu, 29 Aug 2024 13:01:42 GMT
• Title: LLMs vs Established Text Augmentation Techniques for Classification: When do the Benefits Outweight the Costs?
• Authors: Jan Cegin, Jakub Simko, Peter Brusilovsky,
• Abstract summary: We compare effects of recent LLM augmentation methods with established ones on 6 datasets, 3 classifiers and 2 fine-tuning methods. We show that LLM-based methods are worthy of deployment only when very small number of seeds is used.
• Score: 2.7820774076399957
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The generative large language models (LLMs) are increasingly being used for data augmentation tasks, where text samples are LLM-paraphrased and then used for classifier fine-tuning. However, a research that would confirm a clear cost-benefit advantage of LLMs over more established augmentation methods is largely missing. To study if (and when) is the LLM-based augmentation advantageous, we compared the effects of recent LLM augmentation methods with established ones on 6 datasets, 3 classifiers and 2 fine-tuning methods. We also varied the number of seeds and collected samples to better explore the downstream model accuracy space. Finally, we performed a cost-benefit analysis and show that LLM-based methods are worthy of deployment only when very small number of seeds is used. Moreover, in many cases, established methods lead to similar or better model accuracies.

Related papers

• Using Large Language Models for Expert Prior Elicitation in Predictive Modelling [53.54623137152208]
  This study proposes using large language models (LLMs) to elicit expert prior distributions for predictive models. We compare LLM-elicited and uninformative priors, evaluate whether LLMs truthfully generate parameter distributions, and propose a model selection strategy for in-context learning and prior elicitation. Our findings show that LLM-elicited prior parameter distributions significantly reduce predictive error compared to uninformative priors in low-data settings.
  arXiv  Detail & Related papers  (2024-11-26T10:13:39Z)
• Grounding by Trying: LLMs with Reinforcement Learning-Enhanced Retrieval [55.63711219190506]
  Large language models (LLMs) often struggle with posing the right search queries. We introduce $underlineLe$arning to $underlineRe$trieve by $underlineT$rying (LeReT) LeReT can improve the absolute retrieval accuracy by up to 29% and the downstream generator evaluations by 17%.
  arXiv  Detail & Related papers  (2024-10-30T17:02:54Z)
• In-Context Learning with Reinforcement Learning for Incomplete Utterance Rewriting [33.89176174108559]
  In-context learning of large language models (LLMs) makes predictions only based on instructions augmented with a few examples. Existing example selection methods for ICL utilize sparse or dense retrievers and derive effective performance. We propose our policy-based reinforcement learning framework for example selection (RLS), which consists of a language model (LM) selector and an LLM generator.
  arXiv  Detail & Related papers  (2024-08-23T12:32:12Z)
• Enhancing Discriminative Tasks by Guiding the Pre-trained Language Model with Large Language Model's Experience [4.814313782484443]
  Large Language Models (LLMs) and pre-trained Language Models (LMs) have achieved impressive success on many software engineering tasks. We use LLMs to generate domain-specific data, thereby improving the performance of pre-trained LMs on the target tasks.
  arXiv  Detail & Related papers  (2024-08-16T06:37:59Z)
• Delta-CoMe: Training-Free Delta-Compression with Mixed-Precision for Large Language Models [79.46938238953916]
  Fine-tuning large language models (LLMs) to diverse applications is crucial to meet complex demands. Recent studies suggest decomposing a fine-tuned LLM into a base model and corresponding delta weights, which are then compressed using low-rank or low-bit approaches to reduce costs. In this work, we observe that existing low-rank and low-bit compression methods can significantly harm the model performance for task-specific fine-tuned LLMs.
  arXiv  Detail & Related papers  (2024-06-13T07:57:27Z)
• LLMEmbed: Rethinking Lightweight LLM's Genuine Function in Text Classification [13.319594321038926]
  We propose a simple and effective transfer learning strategy, namely LLMEmbed, to address this classical but challenging task. We perform extensive experiments on publicly available datasets, and the results show that LLMEmbed achieves strong performance while enjoys low training overhead.
  arXiv  Detail & Related papers  (2024-06-06T03:46:59Z)
• ReMoDetect: Reward Models Recognize Aligned LLM's Generations [55.06804460642062]
  Large language models (LLMs) generate human-preferable texts. In this paper, we identify the common characteristics shared by these models. We propose two training schemes to further improve the detection ability of the reward model.
  arXiv  Detail & Related papers  (2024-05-27T17:38:33Z)
• LLM2LLM: Boosting LLMs with Novel Iterative Data Enhancement [79.31084387589968]
  Pretrained large language models (LLMs) are currently state-of-the-art for solving the vast majority of natural language processing tasks. We propose LLM2LLM, a data augmentation strategy that uses a teacher LLM to enhance a small seed dataset. We achieve improvements up to 24.2% on the GSM8K dataset, 32.6% on CaseHOLD, 32.0% on SNIPS, 52.6% on TREC and 39.8% on SST-2 over regular fine-tuning in the low-data regime.
  arXiv  Detail & Related papers  (2024-03-22T08:57:07Z)
• Supervised Knowledge Makes Large Language Models Better In-context Learners [94.89301696512776]
  Large Language Models (LLMs) exhibit emerging in-context learning abilities through prompt engineering. The challenge of improving the generalizability and factuality of LLMs in natural language understanding and question answering remains under-explored. We propose a framework that enhances the reliability of LLMs as it: 1) generalizes out-of-distribution data, 2) elucidates how LLMs benefit from discriminative models, and 3) minimizes hallucinations in generative tasks.
  arXiv  Detail & Related papers  (2023-12-26T07:24:46Z)
• LLM-augmented Preference Learning from Natural Language [19.700169351688768]
  Large Language Models (LLMs) are equipped to deal with larger context lengths. LLMs can consistently outperform the SotA when the target text is large. Few-shot learning yields better performance than zero-shot learning.
  arXiv  Detail & Related papers  (2023-10-12T17:17:27Z)
• Scaling Sentence Embeddings with Large Language Models [43.19994568210206]
  In this work, we propose an in-context learning-based method aimed at improving sentence embeddings performance. Our approach involves adapting the previous prompt-based representation method for autoregressive models. By scaling model size, we find scaling to more than tens of billion parameters harms the performance on semantic textual similarity tasks.
  arXiv  Detail & Related papers  (2023-07-31T13:26:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.