Exploring Dataset-Scale Indicators of Data Quality

• URL: http://arxiv.org/abs/2311.04016v1
• Date: Tue, 7 Nov 2023 14:14:32 GMT
• Title: Exploring Dataset-Scale Indicators of Data Quality
• Authors: Benjamin Feuer, Chinmay Hegde
• Abstract summary: Modern computer vision foundation models are trained on massive amounts of data, incurring large economic and environmental costs. Recent research has suggested that improving data quality can significantly reduce the need for data quantity. We posit that the quality of a given dataset can be decomposed into distinct sample-level and dataset-level constituents.
• Score: 23.017200605976807
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Modern computer vision foundation models are trained on massive amounts of data, incurring large economic and environmental costs. Recent research has suggested that improving data quality can significantly reduce the need for data quantity. But what constitutes data quality in computer vision? We posit that the quality of a given dataset can be decomposed into distinct sample-level and dataset-level constituents, and that the former have been more extensively studied than the latter. We ablate the effects of two important dataset-level constituents: label set design, and class balance. By monitoring these constituents using key indicators we provide, researchers and practitioners can better anticipate model performance, measured in terms of its accuracy and robustness to distribution shifts.

Related papers

• A CLIP-Powered Framework for Robust and Generalizable Data Selection [51.46695086779598]
  Real-world datasets often contain redundant and noisy data, imposing a negative impact on training efficiency and model performance. Data selection has shown promise in identifying the most representative samples from the entire dataset. We propose a novel CLIP-powered data selection framework that leverages multimodal information for more robust and generalizable sample selection.
  arXiv  Detail & Related papers  (2024-10-15T03:00:58Z)
• Enhancing Data Quality in Federated Fine-Tuning of Foundation Models [54.757324343062734]
  We propose a data quality control pipeline for federated fine-tuning of foundation models. This pipeline computes scores reflecting the quality of training data and determines a global threshold for a unified standard. Our experiments show that the proposed quality control pipeline facilitates the effectiveness and reliability of the model training, leading to better performance.
  arXiv  Detail & Related papers  (2024-03-07T14:28:04Z)
• Data-Centric Long-Tailed Image Recognition [49.90107582624604]
  Long-tail models exhibit a strong demand for high-quality data. Data-centric approaches aim to enhance both the quantity and quality of data to improve model performance. There is currently a lack of research into the underlying mechanisms explaining the effectiveness of information augmentation.
  arXiv  Detail & Related papers  (2023-11-03T06:34:37Z)
• Assessing Dataset Quality Through Decision Tree Characteristics in Autoencoder-Processed Spaces [0.30458514384586394]
  We show the profound impact of dataset quality on model training and performance. Our findings underscore the importance of appropriate feature selection, adequate data volume, and data quality. This research offers valuable insights into data assessment practices, contributing to the development of more accurate and robust machine learning models.
  arXiv  Detail & Related papers  (2023-06-27T11:33:31Z)
• Striving for data-model efficiency: Identifying data externalities on group performance [75.17591306911015]
  Building trustworthy, effective, and responsible machine learning systems hinges on understanding how differences in training data and modeling decisions interact to impact predictive performance. We focus on a particular type of data-model inefficiency, in which adding training data from some sources can actually lower performance evaluated on key sub-groups of the population. Our results indicate that data-efficiency is a key component of both accurate and trustworthy machine learning.
  arXiv  Detail & Related papers  (2022-11-11T16:48:27Z)
• A Survey of Dataset Refinement for Problems in Computer Vision Datasets [11.45536223418548]
  Large-scale datasets have played a crucial role in the advancement of computer vision. They often suffer from problems such as class imbalance, noisy labels, dataset bias, or high resource costs. Various data-centric solutions have been proposed to solve the dataset problems. They improve the quality of datasets by re-organizing them, which we call dataset refinement.
  arXiv  Detail & Related papers  (2022-10-21T03:58:43Z)
• How Much More Data Do I Need? Estimating Requirements for Downstream Tasks [99.44608160188905]
  Given a small training data set and a learning algorithm, how much more data is necessary to reach a target validation or test performance? Overestimating or underestimating data requirements incurs substantial costs that could be avoided with an adequate budget. Using our guidelines, practitioners can accurately estimate data requirements of machine learning systems to gain savings in both development time and data acquisition costs.
  arXiv  Detail & Related papers  (2022-07-04T21:16:05Z)
• A Proposal to Study "Is High Quality Data All We Need?" [8.122270502556374]
  We propose an empirical study that examines how to select a subset of and/or create high quality benchmark data. We seek to answer if big datasets are truly needed to learn a task, and whether a smaller subset of high quality data can replace big datasets.
  arXiv  Detail & Related papers  (2022-03-12T10:50:13Z)
• On The State of Data In Computer Vision: Human Annotations Remain Indispensable for Developing Deep Learning Models [0.0]
  High-quality labeled datasets play a crucial role in fueling the development of machine learning (ML) Since the emergence of the ImageNet dataset and the AlexNet model in 2012, the size of new open-source labeled vision datasets has remained roughly constant. Only a minority of publications in the computer vision community tackle supervised learning on datasets that are orders of magnitude larger than Imagenet.
  arXiv  Detail & Related papers  (2021-07-31T00:08:21Z)
• Negative Data Augmentation [127.28042046152954]
  We show that negative data augmentation samples provide information on the support of the data distribution. We introduce a new GAN training objective where we use NDA as an additional source of synthetic data for the discriminator. Empirically, models trained with our method achieve improved conditional/unconditional image generation along with improved anomaly detection capabilities.
  arXiv  Detail & Related papers  (2021-02-09T20:28:35Z)
• Data Quality Measures and Efficient Evaluation Algorithms for Large-Scale High-Dimensional Data [0.15229257192293197]
  We propose two data quality measures that can compute class separability and in-class variability, the two important aspects of data quality, for a given dataset. We provide efficient algorithms to compute our quality measures based on random projections and bootstrapping with statistical benefits on large-scale high-dimensional data.
  arXiv  Detail & Related papers  (2021-01-05T10:23:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.