Retrieval-Augmented Data Augmentation for Low-Resource Domain Tasks
- URL: http://arxiv.org/abs/2402.13482v1
- Date: Wed, 21 Feb 2024 02:45:46 GMT
- Title: Retrieval-Augmented Data Augmentation for Low-Resource Domain Tasks
- Authors: Minju Seo, Jinheon Baek, James Thorne, Sung Ju Hwang
- Abstract summary: In low-resource settings, the amount of seed data samples to use for data augmentation is very small.
We propose a novel method that augments training data by incorporating a wealth of examples from other datasets.
This approach can ensure that the generated data is not only relevant but also more diverse than what could be achieved using the limited seed data alone.
- Score: 66.87070857705994
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Despite large successes of recent language models on diverse tasks, they
suffer from severe performance degeneration in low-resource settings with
limited training data available. Many existing works tackle this problem by
generating synthetic data from the training data and then training models on
them, recently using Large Language Models (LLMs). However, in low-resource
settings, the amount of seed data samples to use for data augmentation is very
small, which makes generated samples suboptimal and less diverse. To tackle
this challenge, we propose a novel method that augments training data by
incorporating a wealth of examples from other datasets, along with the given
training data. Specifically, we first retrieve the relevant instances from
other datasets, such as their input-output pairs or contexts, based on their
similarities with the given seed data, and then prompt LLMs to generate new
samples with the contextual information within and across the original and
retrieved samples. This approach can ensure that the generated data is not only
relevant but also more diverse than what could be achieved using the limited
seed data alone. We validate our proposed Retrieval-Augmented Data Augmentation
(RADA) framework on multiple datasets under low-resource settings of training
and test-time data augmentation scenarios, on which it outperforms existing
LLM-powered data augmentation baselines.
Related papers
- Generating Realistic Tabular Data with Large Language Models [49.03536886067729]
Large language models (LLM) have been used for diverse tasks, but do not capture the correct correlation between the features and the target variable.
We propose a LLM-based method with three important improvements to correctly capture the ground-truth feature-class correlation in the real data.
Our experiments show that our method significantly outperforms 10 SOTA baselines on 20 datasets in downstream tasks.
arXiv Detail & Related papers (2024-10-29T04:14:32Z) - Adapt-$\infty$: Scalable Lifelong Multimodal Instruction Tuning via Dynamic Data Selection [89.42023974249122]
Adapt-$infty$ is a new multi-way and adaptive data selection approach for Lifelong Instruction Tuning.
We construct pseudo-skill clusters by grouping gradient-based sample vectors.
We select the best-performing data selector for each skill cluster from a pool of selector experts.
arXiv Detail & Related papers (2024-10-14T15:48:09Z) - FuseGen: PLM Fusion for Data-generation based Zero-shot Learning [18.51772808242954]
FuseGen is a novel data generation-based zero-shot learning framework.
It introduces a new criteria for subset selection from synthetic datasets.
The chosen subset provides in-context feedback to each PLM, enhancing dataset quality.
arXiv Detail & Related papers (2024-06-18T11:55:05Z) - RECOST: External Knowledge Guided Data-efficient Instruction Tuning [25.985023475991625]
We argue that most current data-efficient instruction-tuning methods are highly dependent on the quality of the original instruction-tuning dataset.
We propose a framework dubbed as textbfRECOST, which integrates external-knowledge-base re-ranking and diversity-consistent sampling into a single pipeline.
arXiv Detail & Related papers (2024-02-27T09:47:36Z) - How to Train Data-Efficient LLMs [56.41105687693619]
We study data-efficient approaches for pre-training language models (LLMs)
We find that Ask-LLM and Density sampling are the best methods in their respective categories.
In our comparison of 19 samplers, involving hundreds of evaluation tasks and pre-training runs, we find that Ask-LLM and Density are the best methods in their respective categories.
arXiv Detail & Related papers (2024-02-15T02:27:57Z) - STAR: Boosting Low-Resource Information Extraction by Structure-to-Text
Data Generation with Large Language Models [56.27786433792638]
STAR is a data generation method that leverages Large Language Models (LLMs) to synthesize data instances.
We design fine-grained step-by-step instructions to obtain the initial data instances.
Our experiments show that the data generated by STAR significantly improve the performance of low-resource event extraction and relation extraction tasks.
arXiv Detail & Related papers (2023-05-24T12:15:19Z) - ProGen: Progressive Zero-shot Dataset Generation via In-context Feedback [21.168991554983815]
We propose a progressive zero-shot dataset generation framework, ProGen, to guide the generation of new training data.
We show ProGen achieves on-par or superior performance with only 1% synthetic dataset size.
arXiv Detail & Related papers (2022-10-22T02:07:10Z) - DeGAN : Data-Enriching GAN for Retrieving Representative Samples from a
Trained Classifier [58.979104709647295]
We bridge the gap between the abundance of available data and lack of relevant data, for the future learning tasks of a trained network.
We use the available data, that may be an imbalanced subset of the original training dataset, or a related domain dataset, to retrieve representative samples.
We demonstrate that data from a related domain can be leveraged to achieve state-of-the-art performance.
arXiv Detail & Related papers (2019-12-27T02:05:45Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.