A Bitter Lesson for Data Filtering

• URL: http://arxiv.org/abs/2605.19407v1
• Date: Tue, 19 May 2026 06:02:36 GMT
• Title: A Bitter Lesson for Data Filtering
• Abstract summary: We find that sufficiently trained large parameter models not only tolerate low-quality and distractor data, but in fact benefit from nominally poor'' data.
• Score: 50.6467354069187
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We investigate data filtering for large model pretraining via new scaling studies that target the high compute, data-scarce regime. In spite of an apparently common belief that filtering data to include only high-quality information is essential, our experiments suggest that with enough compute, the best data filter is no data filter. We find that sufficiently trained large parameter models not only tolerate low-quality and distractor data, but in fact benefit from nominally ``poor'' data.

Related papers

• Understanding the Gain from Data Filtering in Multimodal Contrastive Learning [58.09454096371477]
  We show the success of filtered contrastive learning under the standard bimodal data generation model.<n>We show that the error with teacher-based filtering is upper bounded by $frac1sqrtn$ in the large $$ regime, and by $frac1sqrtn$ in the small $$ regime.
  arXiv  Detail & Related papers  (2025-12-16T09:28:38Z)
• LARP: Learner-Agnostic Robust Data Prefiltering [5.530212768657544]
  We formalize the problem of Learner-Agnostic Robust data Prefiltering (LARP)<n>Our theoretical results indicate that performing LARP on a heterogeneous set of learners leads to some loss in model performance.<n>We explore the resulting utility loss and its dependence on the problem parameters.
  arXiv  Detail & Related papers  (2025-06-25T16:07:59Z)
• Who's in and who's out? A case study of multimodal CLIP-filtering in DataComp [13.749279800238092]
  We show that image-text data filtering has biases and is value-laden. Data relating to several imputed demographic groups are associated with higher rates of exclusion. Our conclusions point to a need for fundamental changes in dataset creation and filtering practices.
  arXiv  Detail & Related papers  (2024-05-13T21:53:06Z)
• From Data Deluge to Data Curation: A Filtering-WoRA Paradigm for Efficient Text-based Person Search [30.88999109835329]
  In text-based person search endeavors, data generation has emerged as a prevailing practice, addressing concerns over privacy preservation and the arduous task of manual annotation.<n>We observe that only a subset of the data in constructed datasets plays a decisive role.<n>We introduce a new Filtering-WoRA paradigm, which contains a filtering algorithm to identify this crucial data subset and WoRA learning strategy for light fine-tuning.
  arXiv  Detail & Related papers  (2024-04-16T05:29:14Z)
• Scaling Laws for Data Filtering -- Data Curation cannot be Compute Agnostic [99.3682210827572]
  Vision-language models (VLMs) are trained for thousands of GPU hours on carefully curated web datasets. Data curation strategies are typically developed agnostic of the available compute for training. We introduce neural scaling laws that account for the non-homogeneous nature of web data.
  arXiv  Detail & Related papers  (2024-04-10T17:27:54Z)
• Data Filtering Networks [67.827994353269]
  We study the problem of learning a data filtering network (DFN) for this second step of filtering a large uncurated dataset. Our key finding is that the quality of a network for filtering is distinct from its performance on downstream tasks. Based on our insights, we construct new data filtering networks that induce state-of-the-art image-text datasets.
  arXiv  Detail & Related papers  (2023-09-29T17:37:29Z)
• A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicity [84.6421260559093]
  This study is the largest set of experiments to validate, quantify, and expose undocumented intuitions about text pretraining. Our findings indicate there does not exist a one-size-fits-all solution to filtering training data.
  arXiv  Detail & Related papers  (2023-05-22T15:57:53Z)
• An Empirical Exploration in Quality Filtering of Text Data [0.0]
  We find that aggressive filtering can lead to a decrease in model quality on a wide array of downstream tasks for a GPT-like language model. We speculate that this is because optimizing sufficiently strongly for a proxy metric harms performance on the true objective.
  arXiv  Detail & Related papers  (2021-09-02T04:02:51Z)
• Adversarial Filters of Dataset Biases [96.090959788952]
  Large neural models have demonstrated human-level performance on language and vision benchmarks. Their performance degrades considerably on adversarial or out-of-distribution samples. We propose AFLite, which adversarially filters such dataset biases.
  arXiv  Detail & Related papers  (2020-02-10T21:59:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.