A Survey on Data Selection for Language Models
- URL: http://arxiv.org/abs/2402.16827v2
- Date: Fri, 8 Mar 2024 20:04:01 GMT
- Title: A Survey on Data Selection for Language Models
- Authors: Alon Albalak, Yanai Elazar, Sang Michael Xie, Shayne Longpre, Nathan
Lambert, Xinyi Wang, Niklas Muennighoff, Bairu Hou, Liangming Pan, Haewon
Jeong, Colin Raffel, Shiyu Chang, Tatsunori Hashimoto, William Yang Wang
- Abstract summary: Data selection methods aim to determine which data points to include in a training dataset.
Deep learning is mostly driven by empirical evidence and experimentation on large-scale data is expensive.
Few organizations have the resources for extensive data selection research.
- Score: 151.6210632830082
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: A major factor in the recent success of large language models is the use of
enormous and ever-growing text datasets for unsupervised pre-training. However,
naively training a model on all available data may not be optimal (or
feasible), as the quality of available text data can vary. Filtering out data
can also decrease the carbon footprint and financial costs of training models
by reducing the amount of training required. Data selection methods aim to
determine which candidate data points to include in the training dataset and
how to appropriately sample from the selected data points. The promise of
improved data selection methods has caused the volume of research in the area
to rapidly expand. However, because deep learning is mostly driven by empirical
evidence and experimentation on large-scale data is expensive, few
organizations have the resources for extensive data selection research.
Consequently, knowledge of effective data selection practices has become
concentrated within a few organizations, many of which do not openly share
their findings and methodologies. To narrow this gap in knowledge, we present a
comprehensive review of existing literature on data selection methods and
related research areas, providing a taxonomy of existing approaches. By
describing the current landscape of research, this work aims to accelerate
progress in data selection by establishing an entry point for new and
established researchers. Additionally, throughout this review we draw attention
to noticeable holes in the literature and conclude the paper by proposing
promising avenues for future research.
Related papers
- A Survey on Data Synthesis and Augmentation for Large Language Models [35.59526251210408]
This paper reviews and summarizes data generation techniques throughout the lifecycle of Large Language Models.
We discuss the current constraints faced by these methods and investigate potential pathways for future development and research.
arXiv Detail & Related papers (2024-10-16T16:12:39Z) - Unsupervised Data Validation Methods for Efficient Model Training [0.0]
State-of-the-art models in natural language processing (NLP), text-to-speech (TTS), speech-to-text (STT) and vision-language models (VLM) rely heavily on large datasets.
This research explores key areas such as defining "quality data," developing methods for generating appropriate data and enhancing accessibility to model training.
arXiv Detail & Related papers (2024-10-10T13:00:53Z) - Unleashing the Power of Data Tsunami: A Comprehensive Survey on Data Assessment and Selection for Instruction Tuning of Language Models [33.488331159912136]
Instruction tuning plays a critical role in aligning large language models (LLMs) with human preference.
Data assessment and selection methods have been proposed in the fields of natural language processing (NLP) and deep learning.
We present a comprehensive review on existing literature of data assessment and selection especially for instruction tuning of LLMs.
arXiv Detail & Related papers (2024-08-04T16:50:07Z) - Data-Centric AI in the Age of Large Language Models [51.20451986068925]
This position paper proposes a data-centric viewpoint of AI research, focusing on large language models (LLMs)
We make the key observation that data is instrumental in the developmental (e.g., pretraining and fine-tuning) and inferential stages (e.g., in-context learning) of LLMs.
We identify four specific scenarios centered around data, covering data-centric benchmarks and data curation, data attribution, knowledge transfer, and inference contextualization.
arXiv Detail & Related papers (2024-06-20T16:34:07Z) - LESS: Selecting Influential Data for Targeted Instruction Tuning [64.78894228923619]
We propose LESS, an efficient algorithm to estimate data influences and perform Low-rank gradiEnt Similarity Search for instruction data selection.
We show that training on a LESS-selected 5% of the data can often outperform training on the full dataset across diverse downstream tasks.
Our method goes beyond surface form cues to identify data that the necessary reasoning skills for the intended downstream application.
arXiv Detail & Related papers (2024-02-06T19:18:04Z) - Data Optimization in Deep Learning: A Survey [3.1274367448459253]
This study aims to organize a wide range of existing data optimization methodologies for deep learning.
The constructed taxonomy considers the diversity of split dimensions, and deep sub-taxonomies are constructed for each dimension.
The constructed taxonomy and the revealed connections will enlighten the better understanding of existing methods and the design of novel data optimization techniques.
arXiv Detail & Related papers (2023-10-25T09:33:57Z) - Exploring Data Redundancy in Real-world Image Classification through
Data Selection [20.389636181891515]
Deep learning models often require large amounts of data for training, leading to increased costs.
We present two data valuation metrics based on Synaptic Intelligence and gradient norms, respectively, to study redundancy in real-world image data.
Online and offline data selection algorithms are then proposed via clustering and grouping based on the examined data values.
arXiv Detail & Related papers (2023-06-25T03:31:05Z) - A Survey of Learning on Small Data: Generalization, Optimization, and
Challenge [101.27154181792567]
Learning on small data that approximates the generalization ability of big data is one of the ultimate purposes of AI.
This survey follows the active sampling theory under a PAC framework to analyze the generalization error and label complexity of learning on small data.
Multiple data applications that may benefit from efficient small data representation are surveyed.
arXiv Detail & Related papers (2022-07-29T02:34:19Z) - Research Trends and Applications of Data Augmentation Algorithms [77.34726150561087]
We identify the main areas of application of data augmentation algorithms, the types of algorithms used, significant research trends, their progression over time and research gaps in data augmentation literature.
We expect readers to understand the potential of data augmentation, as well as identify future research directions and open questions within data augmentation research.
arXiv Detail & Related papers (2022-07-18T11:38:32Z) - An Empirical Survey of Data Augmentation for Limited Data Learning in
NLP [88.65488361532158]
dependence on abundant data prevents NLP models from being applied to low-resource settings or novel tasks.
Data augmentation methods have been explored as a means of improving data efficiency in NLP.
We provide an empirical survey of recent progress on data augmentation for NLP in the limited labeled data setting.
arXiv Detail & Related papers (2021-06-14T15:27:22Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.