A Survey on Data Quality Dimensions and Tools for Machine Learning

• URL: http://arxiv.org/abs/2406.19614v1
• Date: Fri, 28 Jun 2024 02:41:33 GMT
• Title: A Survey on Data Quality Dimensions and Tools for Machine Learning
• Authors: Yuhan Zhou, Fengjiao Tu, Kewei Sha, Junhua Ding, Haihua Chen,
• Abstract summary: Machine learning (ML) technologies have become substantial in practically all aspects of our society. Traditional methods like exploratory data analysis (EDA) and cross-validation (CV) face challenges in data-centric AI. In this survey, we review 17 DQ evaluation and improvement tools in the last 5 years.
• Score: 1.6078134198754157
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Machine learning (ML) technologies have become substantial in practically all aspects of our society, and data quality (DQ) is critical for the performance, fairness, robustness, safety, and scalability of ML models. With the large and complex data in data-centric AI, traditional methods like exploratory data analysis (EDA) and cross-validation (CV) face challenges, highlighting the importance of mastering DQ tools. In this survey, we review 17 DQ evaluation and improvement tools in the last 5 years. By introducing the DQ dimensions, metrics, and main functions embedded in these tools, we compare their strengths and limitations and propose a roadmap for developing open-source DQ tools for ML. Based on the discussions on the challenges and emerging trends, we further highlight the potential applications of large language models (LLMs) and generative AI in DQ evaluation and improvement for ML. We believe this comprehensive survey can enhance understanding of DQ in ML and could drive progress in data-centric AI. A complete list of the literature investigated in this survey is available on GitHub at: https://github.com/haihua0913/awesome-dq4ml.

Related papers

• SIaM: Self-Improving Code-Assisted Mathematical Reasoning of Large Language Models [54.78329741186446]
  We propose a novel paradigm that uses a code-based critic model to guide steps including question-code data construction, quality control, and complementary evaluation. Experiments across both in-domain and out-of-domain benchmarks in English and Chinese demonstrate the effectiveness of the proposed paradigm.
  arXiv  Detail & Related papers  (2024-08-28T06:33:03Z)
• Benchmarking Uncertainty Quantification Methods for Large Language Models with LM-Polygraph [83.90988015005934]
  Uncertainty quantification (UQ) is a critical component of machine learning (ML) applications. We introduce a novel benchmark that implements a collection of state-of-the-art UQ baselines. We conduct a large-scale empirical investigation of UQ and normalization techniques across nine tasks, and identify the most promising approaches.
  arXiv  Detail & Related papers  (2024-06-21T20:06:31Z)
• Data Quality in Edge Machine Learning: A State-of-the-Art Survey [2.8449839307925955]
  Data-driven Artificial Intelligence (AI) systems trained using Machine Learning (ML) are shaping an ever-increasing portion of our lives. On the one hand, the outsized influence of these systems imposes a high standard of quality, particularly in the data used to train them. On the other hand, establishing and maintaining standards of Data Quality (DQ) becomes more challenging due to the proliferation of Edge Computing and Internet of Things devices.
  arXiv  Detail & Related papers  (2024-06-01T23:07:05Z)
• Synthesize Step-by-Step: Tools, Templates and LLMs as Data Generators for Reasoning-Based Chart VQA [9.659820850719413]
  We leverage Large Language Models (LLMs), which have shown to have strong reasoning ability, as an automatic data annotator. Key innovation in our method lies in the Synthesize Step-by-Step strategy. We significantly enhance the chart VQA models, achieving the state-of-the-art accuracy on the ChartQA and PlotQA datasets.
  arXiv  Detail & Related papers  (2024-03-25T03:02:27Z)
• Automatic Question-Answer Generation for Long-Tail Knowledge [65.11554185687258]
  We propose an automatic approach to generate specialized QA datasets for tail entities. We conduct extensive experiments by employing pretrained LLMs on our newly generated long-tail QA datasets.
  arXiv  Detail & Related papers  (2024-03-03T03:06:31Z)
• AQUALLM: Audio Question Answering Data Generation Using Large Language Models [2.2232550112727267]
  We introduce a scalable AQA data generation pipeline, which relies on Large Language Models (LLMs) We present three extensive and high-quality benchmark datasets for AQA. Models trained on our datasets demonstrate enhanced generalizability when compared to models trained using human-annotated AQA data.
  arXiv  Detail & Related papers  (2023-12-28T20:01:27Z)
• QASnowball: An Iterative Bootstrapping Framework for High-Quality Question-Answering Data Generation [67.27999343730224]
  We introduce an iterative bootstrapping framework for QA data augmentation (named QASnowball) QASnowball can iteratively generate large-scale high-quality QA data based on a seed set of supervised examples. We conduct experiments in the high-resource English scenario and the medium-resource Chinese scenario, and the experimental results show that the data generated by QASnowball can facilitate QA models.
  arXiv  Detail & Related papers  (2023-09-19T05:20:36Z)
• Data-centric Artificial Intelligence: A Survey [47.24049907785989]
  Recently, the role of data in AI has been significantly magnified, giving rise to the emerging concept of data-centric AI. In this survey, we discuss the necessity of data-centric AI, followed by a holistic view of three general data-centric goals. We believe this is the first comprehensive survey that provides a global view of a spectrum of tasks across various stages of the data lifecycle.
  arXiv  Detail & Related papers  (2023-03-17T17:44:56Z)
• DC-Check: A Data-Centric AI checklist to guide the development of reliable machine learning systems [81.21462458089142]
  Data-centric AI is emerging as a unifying paradigm that could enable reliable end-to-end pipelines. We propose DC-Check, an actionable checklist-style framework to elicit data-centric considerations. This data-centric lens on development aims to promote thoughtfulness and transparency prior to system development.
  arXiv  Detail & Related papers  (2022-11-09T17:32:09Z)
• A New Tool for Efficiently Generating Quality Estimation Datasets [1.1374578778690623]
  Building data for quality estimation (QE) training is expensive and requires significant human labor. We propose a fully automatic pseudo-QE dataset generation tool that generates QE datasets by receiving only monolingual or parallel corpus as the input.
  arXiv  Detail & Related papers  (2021-11-01T08:37:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.