How to Train Data-Efficient LLMs
- URL: http://arxiv.org/abs/2402.09668v1
- Date: Thu, 15 Feb 2024 02:27:57 GMT
- Title: How to Train Data-Efficient LLMs
- Authors: Noveen Sachdeva, Benjamin Coleman, Wang-Cheng Kang, Jianmo Ni, Lichan
Hong, Ed H. Chi, James Caverlee, Julian McAuley, Derek Zhiyuan Cheng
- Abstract summary: We study data-efficient approaches for pre-training language models (LLMs)
We find that Ask-LLM and Density sampling are the best methods in their respective categories.
In our comparison of 19 samplers, involving hundreds of evaluation tasks and pre-training runs, we find that Ask-LLM and Density are the best methods in their respective categories.
- Score: 56.41105687693619
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: The training of large language models (LLMs) is expensive. In this paper, we
study data-efficient approaches for pre-training LLMs, i.e., techniques that
aim to optimize the Pareto frontier of model quality and training resource/data
consumption. We seek to understand the tradeoffs associated with data selection
routines based on (i) expensive-to-compute data-quality estimates, and (ii)
maximization of coverage and diversity-based measures in the feature space. Our
first technique, Ask-LLM, leverages the zero-shot reasoning capabilities of
instruction-tuned LLMs to directly assess the quality of a training example. To
target coverage, we propose Density sampling, which models the data
distribution to select a diverse sample. In our comparison of 19 samplers,
involving hundreds of evaluation tasks and pre-training runs, we find that
Ask-LLM and Density are the best methods in their respective categories.
Coverage sampling can recover the performance of the full data, while models
trained on Ask-LLM data consistently outperform full-data training -- even when
we reject 90% of the original dataset, while converging up to 70% faster.
Related papers
- Entropy Law: The Story Behind Data Compression and LLM Performance [115.70395740286422]
We find that model performance is negatively correlated to the compression ratio of training data, which usually yields a lower training loss.
Based on the findings of the entropy law, we propose a quite efficient and universal data selection method.
We also present an interesting application of entropy law that can detect potential performance risks at the beginning of model training.
arXiv Detail & Related papers (2024-07-09T08:14:29Z) - Uncertainty Aware Learning for Language Model Alignment [97.36361196793929]
We propose uncertainty-aware learning (UAL) to improve the model alignment of different task scenarios.
We implement UAL in a simple fashion -- adaptively setting the label smoothing value of training according to the uncertainty of individual samples.
Experiments on widely used benchmarks demonstrate that our UAL significantly and consistently outperforms standard supervised fine-tuning.
arXiv Detail & Related papers (2024-06-07T11:37:45Z) - Take the Bull by the Horns: Hard Sample-Reweighted Continual Training
Improves LLM Generalization [165.98557106089777]
A key challenge is to enhance the capabilities of large language models (LLMs) amid a looming shortage of high-quality training data.
Our study starts from an empirical strategy for the light continual training of LLMs using their original pre-training data sets.
We then formalize this strategy into a principled framework of Instance-Reweighted Distributionally Robust Optimization.
arXiv Detail & Related papers (2024-02-22T04:10:57Z) - DsDm: Model-Aware Dataset Selection with Datamodels [81.01744199870043]
Standard practice is to filter for examples that match human notions of data quality.
We find that selecting according to similarity with "high quality" data sources may not increase (and can even hurt) performance compared to randomly selecting data.
Our framework avoids handpicked notions of data quality, and instead models explicitly how the learning process uses train datapoints to predict on the target tasks.
arXiv Detail & Related papers (2024-01-23T17:22:00Z) - Efficient Online Data Mixing For Language Model Pre-Training [101.45242332613944]
Existing data selection methods suffer from slow and computationally expensive processes.
Data mixing, on the other hand, reduces the complexity of data selection by grouping data points together.
We develop an efficient algorithm for Online Data Mixing (ODM) that combines elements from both data selection and data mixing.
arXiv Detail & Related papers (2023-12-05T00:42:35Z) - When Less is More: Investigating Data Pruning for Pretraining LLMs at
Scale [12.94829977468838]
Large volumes of text data have contributed significantly to the development of large language models.
To date, efforts to prune datasets down to a higher quality subset have relied on hand-crafteds encoded as rule-based filters.
We take a wider view and explore scalable estimates of data quality that can be used to measure the quality of pretraining data.
arXiv Detail & Related papers (2023-09-08T19:34:05Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.