DataAssist: A Machine Learning Approach to Data Cleaning and Preparation

• URL: http://arxiv.org/abs/2307.07119v2
• Date: Mon, 17 Jul 2023 14:16:05 GMT
• Title: DataAssist: A Machine Learning Approach to Data Cleaning and Preparation
• Authors: Kartikay Goyle, Quin Xie and Vakul Goyle
• Abstract summary: DataAssist is an automated data preparation and cleaning platform that enhances dataset quality using ML-informed methods. Our tool is applicable to a variety of fields, including economics, business, and forecasting applications saving over 50% time of the time spent on data cleansing and preparation.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Current automated machine learning (ML) tools are model-centric, focusing on model selection and parameter optimization. However, the majority of the time in data analysis is devoted to data cleaning and wrangling, for which limited tools are available. Here we present DataAssist, an automated data preparation and cleaning platform that enhances dataset quality using ML-informed methods. We show that DataAssist provides a pipeline for exploratory data analysis and data cleaning, including generating visualization for user-selected variables, unifying data annotation, suggesting anomaly removal, and preprocessing data. The exported dataset can be readily integrated with other autoML tools or user-specified model for downstream analysis. Our data-centric tool is applicable to a variety of fields, including economics, business, and forecasting applications saving over 50% time of the time spent on data cleansing and preparation.

Related papers

• Data Advisor: Dynamic Data Curation for Safety Alignment of Large Language Models [79.65071553905021]
  We propose Data Advisor, a method for generating data that takes into account the characteristics of the desired dataset. Data Advisor monitors the status of the generated data, identifies weaknesses in the current dataset, and advises the next iteration of data generation.
  arXiv  Detail & Related papers  (2024-10-07T17:59:58Z)
• The Frontier of Data Erasure: Machine Unlearning for Large Language Models [56.26002631481726]
  Large Language Models (LLMs) are foundational to AI advancements. LLMs pose risks by potentially memorizing and disseminating sensitive, biased, or copyrighted information. Machine unlearning emerges as a cutting-edge solution to mitigate these concerns.
  arXiv  Detail & Related papers  (2024-03-23T09:26:15Z)
• How to Train Data-Efficient LLMs [56.41105687693619]
  We study data-efficient approaches for pre-training language models (LLMs) We find that Ask-LLM and Density sampling are the best methods in their respective categories. In our comparison of 19 samplers, involving hundreds of evaluation tasks and pre-training runs, we find that Ask-LLM and Density are the best methods in their respective categories.
  arXiv  Detail & Related papers  (2024-02-15T02:27:57Z)
• LESS: Selecting Influential Data for Targeted Instruction Tuning [64.78894228923619]
  We propose LESS, an efficient algorithm to estimate data influences and perform Low-rank gradiEnt Similarity Search for instruction data selection. We show that training on a LESS-selected 5% of the data can often outperform training on the full dataset across diverse downstream tasks. Our method goes beyond surface form cues to identify data that the necessary reasoning skills for the intended downstream application.
  arXiv  Detail & Related papers  (2024-02-06T19:18:04Z)
• Dataset Factory: A Toolchain For Generative Computer Vision Datasets [0.9013233848500058]
  We propose a "dataset factory" that separates the storage and processing of samples from metadata. This enables data-centric operations at scale for machine learning teams and individual researchers.
  arXiv  Detail & Related papers  (2023-09-20T19:43:37Z)
• DataRaceBench V1.4.1 and DataRaceBench-ML V0.1: Benchmark Suites for Data Race Detection [23.240375422302666]
  Data races pose a significant threat in multi-threaded parallel applications due to their negative impact on program correctness. Open-source benchmark suite, DataRaceBench, is crafted to assess these data race detection tools in a systematic and measurable manner. This paper introduces a derived dataset named DataRaceBench-ML (DRB-ML).
  arXiv  Detail & Related papers  (2023-08-16T16:23:13Z)
• Demonstration of InsightPilot: An LLM-Empowered Automated Data Exploration System [48.62158108517576]
  We introduce InsightPilot, an automated data exploration system designed to simplify the data exploration process. InsightPilot automatically selects appropriate analysis intents, such as understanding, summarizing, and explaining. In brief, an IQuery is an abstraction and automation of data analysis operations, which mimics the approach of data analysts.
  arXiv  Detail & Related papers  (2023-04-02T07:27:49Z)
• Designing Data: Proactive Data Collection and Iteration for Machine Learning [12.295169687537395]
  Lack of diversity in data collection has caused significant failures in machine learning (ML) applications. New methods to track & manage data collection, iteration, and model training are necessary for evaluating whether datasets reflect real world variability.
  arXiv  Detail & Related papers  (2023-01-24T21:40:29Z)
• Fix your Models by Fixing your Datasets [0.6058427379240697]
  Current machine learning tools lack streamlined processes for improving the data quality. We introduce a systematic framework for finding noisy or mislabelled samples in the dataset. We demonstrate the efficacy of our framework on public as well as private enterprise datasets of two Fortune 500 companies.
  arXiv  Detail & Related papers  (2021-12-15T02:41:50Z)
• Automated Machine Learning Techniques for Data Streams [91.3755431537592]
  This paper surveys the state-of-the-art open-source AutoML tools, applies them to data collected from streams, and measures how their performance changes over time. The results show that off-the-shelf AutoML tools can provide satisfactory results but in the presence of concept drift, detection or adaptation techniques have to be applied to maintain the predictive accuracy over time.
  arXiv  Detail & Related papers  (2021-06-14T11:42:46Z)
• Improving the Performance of Fine-Grain Image Classifiers via Generative Data Augmentation [0.5161531917413706]
  We develop Data Augmentation from Proficient Pre-Training of Robust Generative Adrial Networks (DAPPER GAN) DAPPER GAN is an ML analytics support tool that automatically generates novel views of training images. We experimentally evaluate this technique on the Stanford Cars dataset, demonstrating improved vehicle make and model classification accuracy.
  arXiv  Detail & Related papers  (2020-08-12T15:29:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.