Increasing Diversity While Maintaining Accuracy: Text Data Generation
with Large Language Models and Human Interventions
- URL: http://arxiv.org/abs/2306.04140v1
- Date: Wed, 7 Jun 2023 04:27:09 GMT
- Title: Increasing Diversity While Maintaining Accuracy: Text Data Generation
with Large Language Models and Human Interventions
- Authors: John Joon Young Chung, Ece Kamar, Saleema Amershi
- Abstract summary: Large language models (LLMs) can be used to generate text data for training and evaluating other models.
We explore human-AI partnerships to facilitate high diversity and accuracy in LLM-based text data generation.
- Score: 30.464763055981933
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Large language models (LLMs) can be used to generate text data for training
and evaluating other models. However, creating high-quality datasets with LLMs
can be challenging. In this work, we explore human-AI partnerships to
facilitate high diversity and accuracy in LLM-based text data generation. We
first examine two approaches to diversify text generation: 1) logit
suppression, which minimizes the generation of languages that have already been
frequently generated, and 2) temperature sampling, which flattens the token
sampling probability. We found that diversification approaches can increase
data diversity but often at the cost of data accuracy (i.e., text and labels
being appropriate for the target domain). To address this issue, we examined
two human interventions, 1) label replacement (LR), correcting misaligned
labels, and 2) out-of-scope filtering (OOSF), removing instances that are out
of the user's domain of interest or to which no considered label applies. With
oracle studies, we found that LR increases the absolute accuracy of models
trained with diversified datasets by 14.4%. Moreover, we found that some models
trained with data generated with LR interventions outperformed LLM-based
few-shot classification. In contrast, OOSF was not effective in increasing
model accuracy, implying the need for future work in human-in-the-loop text
data generation.
Related papers
- DARG: Dynamic Evaluation of Large Language Models via Adaptive Reasoning Graph [70.79413606968814]
We introduce Dynamic Evaluation of LLMs via Adaptive Reasoning Graph Evolvement (DARG) to dynamically extend current benchmarks with controlled complexity and diversity.
Specifically, we first extract the reasoning graphs of data points in current benchmarks and then perturb the reasoning graphs to generate novel testing data.
Such newly generated test samples can have different levels of complexity while maintaining linguistic diversity similar to the original benchmarks.
arXiv Detail & Related papers (2024-06-25T04:27:53Z) - Leveraging Weakly Annotated Data for Hate Speech Detection in Code-Mixed
Hinglish: A Feasibility-Driven Transfer Learning Approach with Large Language
Models [0.0]
Hate speech detection in mix-code low-resource languages is an active problem area where the use of Large Language Models has proven beneficial.
In this study, we have compiled a dataset of 100 YouTube comments, and weakly labelled them for coarse and fine-grained misogyny classification in mix-code Hinglish.
Out of all the approaches, zero-shot classification using the Bidirectional Auto-Regressive Transformers (BART) large model and few-shot prompting using Generative Pre-trained Transformer- 3 (ChatGPT-3) achieve the best results.
arXiv Detail & Related papers (2024-03-04T15:27:49Z) - Retrieval-Augmented Data Augmentation for Low-Resource Domain Tasks [66.87070857705994]
In low-resource settings, the amount of seed data samples to use for data augmentation is very small.
We propose a novel method that augments training data by incorporating a wealth of examples from other datasets.
This approach can ensure that the generated data is not only relevant but also more diverse than what could be achieved using the limited seed data alone.
arXiv Detail & Related papers (2024-02-21T02:45:46Z) - A Simple yet Efficient Ensemble Approach for AI-generated Text Detection [0.5840089113969194]
Large Language Models (LLMs) have demonstrated remarkable capabilities in generating text that closely resembles human writing.
It is essential to build automated approaches capable of distinguishing between artificially generated text and human-authored text.
We propose a simple yet efficient solution by ensembling predictions from multiple constituent LLMs.
arXiv Detail & Related papers (2023-11-06T13:11:02Z) - LLMaAA: Making Large Language Models as Active Annotators [32.57011151031332]
We propose LLMaAA, which takes large language models as annotators and puts them into an active learning loop to determine what to annotate efficiently.
We conduct experiments and analysis on two classic NLP tasks, named entity recognition and relation extraction.
With LLMaAA, task-specific models trained from LLM-generated labels can outperform the teacher within only hundreds of annotated examples.
arXiv Detail & Related papers (2023-10-30T14:54:15Z) - RegaVAE: A Retrieval-Augmented Gaussian Mixture Variational Auto-Encoder
for Language Modeling [79.56442336234221]
We introduce RegaVAE, a retrieval-augmented language model built upon the variational auto-encoder (VAE)
It encodes the text corpus into a latent space, capturing current and future information from both source and target text.
Experimental results on various datasets demonstrate significant improvements in text generation quality and hallucination removal.
arXiv Detail & Related papers (2023-10-16T16:42:01Z) - MLLM-DataEngine: An Iterative Refinement Approach for MLLM [62.30753425449056]
We propose a novel closed-loop system that bridges data generation, model training, and evaluation.
Within each loop, the MLLM-DataEngine first analyze the weakness of the model based on the evaluation results.
For targeting, we propose an Adaptive Bad-case Sampling module, which adjusts the ratio of different types of data.
For quality, we resort to GPT-4 to generate high-quality data with each given data type.
arXiv Detail & Related papers (2023-08-25T01:41:04Z) - Can Large Language Models Infer Causation from Correlation? [104.96351414570239]
We test the pure causal inference skills of large language models (LLMs)
We formulate a novel task Corr2Cause, which takes a set of correlational statements and determines the causal relationship between the variables.
We show that these models achieve almost close to random performance on the task.
arXiv Detail & Related papers (2023-06-09T12:09:15Z) - Improving Classifier Robustness through Active Generation of Pairwise
Counterfactuals [22.916599410472102]
We present a novel framework that utilizes counterfactual generative models to generate a large number of diverse counterfactuals.
We show that with a small amount of human-annotated counterfactual data (10%), we can generate a counterfactual augmentation dataset with learned labels.
arXiv Detail & Related papers (2023-05-22T23:19:01Z) - Self-augmented Data Selection for Few-shot Dialogue Generation [18.794770678708637]
We adopt the self-training framework to deal with the few-shot MR-to-Text generation problem.
We propose a novel data selection strategy to select the data that our generation model is most uncertain about.
arXiv Detail & Related papers (2022-05-19T16:25:50Z) - Improving Classifier Training Efficiency for Automatic Cyberbullying
Detection with Feature Density [58.64907136562178]
We study the effectiveness of Feature Density (FD) using different linguistically-backed feature preprocessing methods.
We hypothesise that estimating dataset complexity allows for the reduction of the number of required experiments.
The difference in linguistic complexity of datasets allows us to additionally discuss the efficacy of linguistically-backed word preprocessing.
arXiv Detail & Related papers (2021-11-02T15:48:28Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.