An Empirical Exploration in Quality Filtering of Text Data
- URL: http://arxiv.org/abs/2109.00698v1
- Date: Thu, 2 Sep 2021 04:02:51 GMT
- Title: An Empirical Exploration in Quality Filtering of Text Data
- Authors: Leo Gao
- Abstract summary: We find that aggressive filtering can lead to a decrease in model quality on a wide array of downstream tasks for a GPT-like language model.
We speculate that this is because optimizing sufficiently strongly for a proxy metric harms performance on the true objective.
- Score: 0.0
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: While conventional wisdom suggests that more aggressively filtering data from
low-quality sources like Common Crawl always monotonically improves the quality
of training data, we find that aggressive filtering can in fact lead to a
decrease in model quality on a wide array of downstream tasks for a GPT-like
language model. We speculate that this is because optimizing sufficiently
strongly for a proxy metric harms performance on the true objective, suggesting
a need for more robust filtering objectives when attempting to filter more
aggressively. We hope this work leads to detailed analysis of the effects of
dataset filtering design choices on downstream model performance in future
work.
Related papers
- ScalingFilter: Assessing Data Quality through Inverse Utilization of Scaling Laws [67.59263833387536]
ScalingFilter is a novel approach that evaluates text quality based on the perplexity difference between two language models trained on the same data.
To assess the bias introduced by quality filtering, we introduce semantic diversity, a metric of utilizing text embedding models for semantic representations.
arXiv Detail & Related papers (2024-08-15T17:59:30Z) - Filtered Direct Preference Optimization [7.060398061192042]
Reinforcement learning from human feedback (RLHF) plays a crucial role in aligning language models with human preferences.
This paper addresses the issue of text quality within the preference dataset by focusing on direct preference optimization (DPO)
We propose an extension of DPO, termed filtered direct preference optimization (fDPO)
arXiv Detail & Related papers (2024-04-22T03:05:19Z) - From Data Deluge to Data Curation: A Filtering-WoRA Paradigm for Efficient Text-based Person Search [19.070305201045954]
In text-based person search endeavors, data generation has emerged as a prevailing practice, addressing concerns over privacy preservation and the arduous task of manual annotation.
We observe that only a subset of the data in constructed datasets plays a decisive role.
We introduce a new Filtering-WoRA paradigm, which contains a filtering algorithm to identify this crucial data subset and WoRA learning strategy for light fine-tuning.
arXiv Detail & Related papers (2024-04-16T05:29:14Z) - Superfiltering: Weak-to-Strong Data Filtering for Fast Instruction-Tuning [43.10197671420528]
We study Superfiltering: Can we use a smaller and weaker model to select data for finetuning a larger and stronger model?
This enables us to use a much smaller and more efficient model to filter the instruction data used to train a larger language model.
Not only does it largely speed up the data filtering, but the filtered-data-finetuned LLM achieves even better performance on standard benchmarks.
arXiv Detail & Related papers (2024-02-01T11:57:53Z) - DsDm: Model-Aware Dataset Selection with Datamodels [81.01744199870043]
Standard practice is to filter for examples that match human notions of data quality.
We find that selecting according to similarity with "high quality" data sources may not increase (and can even hurt) performance compared to randomly selecting data.
Our framework avoids handpicked notions of data quality, and instead models explicitly how the learning process uses train datapoints to predict on the target tasks.
arXiv Detail & Related papers (2024-01-23T17:22:00Z) - Data Filtering Networks [67.827994353269]
We study the problem of learning a data filtering network (DFN) for this second step of filtering a large uncurated dataset.
Our key finding is that the quality of a network for filtering is distinct from its performance on downstream tasks.
Based on our insights, we construct new data filtering networks that induce state-of-the-art image-text datasets.
arXiv Detail & Related papers (2023-09-29T17:37:29Z) - Filter-enhanced MLP is All You Need for Sequential Recommendation [89.0974365344997]
In online platforms, logged user behavior data is inevitable to contain noise.
We borrow the idea of filtering algorithms from signal processing that attenuates the noise in the frequency domain.
We propose textbfFMLP-Rec, an all-MLP model with learnable filters for sequential recommendation task.
arXiv Detail & Related papers (2022-02-28T05:49:35Z) - Dependency Aware Filter Pruning [74.69495455411987]
Pruning a proportion of unimportant filters is an efficient way to mitigate the inference cost.
Previous work prunes filters according to their weight norms or the corresponding batch-norm scaling factors.
We propose a novel mechanism to dynamically control the sparsity-inducing regularization so as to achieve the desired sparsity.
arXiv Detail & Related papers (2020-05-06T07:41:22Z) - Adversarial Filters of Dataset Biases [96.090959788952]
Large neural models have demonstrated human-level performance on language and vision benchmarks.
Their performance degrades considerably on adversarial or out-of-distribution samples.
We propose AFLite, which adversarially filters such dataset biases.
arXiv Detail & Related papers (2020-02-10T21:59:21Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.