DataAssist: A Machine Learning Approach to Data Cleaning and Preparation

• URL: http://arxiv.org/abs/2307.07119v2
• Date: Mon, 17 Jul 2023 14:16:05 GMT
• Title: DataAssist: A Machine Learning Approach to Data Cleaning and Preparation
• Authors: Kartikay Goyle, Quin Xie and Vakul Goyle
• Abstract summary: DataAssist is an automated data preparation and cleaning platform that enhances dataset quality using ML-informed methods. Our tool is applicable to a variety of fields, including economics, business, and forecasting applications saving over 50% time of the time spent on data cleansing and preparation.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Current automated machine learning (ML) tools are model-centric, focusing on model selection and parameter optimization. However, the majority of the time in data analysis is devoted to data cleaning and wrangling, for which limited tools are available. Here we present DataAssist, an automated data preparation and cleaning platform that enhances dataset quality using ML-informed methods. We show that DataAssist provides a pipeline for exploratory data analysis and data cleaning, including generating visualization for user-selected variables, unifying data annotation, suggesting anomaly removal, and preprocessing data. The exported dataset can be readily integrated with other autoML tools or user-specified model for downstream analysis. Our data-centric tool is applicable to a variety of fields, including economics, business, and forecasting applications saving over 50% time of the time spent on data cleansing and preparation.

Related papers

• Towards Data-Efficient Pretraining for Atomic Property Prediction [51.660835328611626]
  We show that pretraining on a task-relevant dataset can match or surpass large-scale pretraining. We introduce the Chemical Similarity Index (CSI), a novel metric inspired by computer vision's Fr'echet Inception Distance.
  arXiv  Detail & Related papers  (2025-02-16T11:46:23Z)
• Large Language Models and Synthetic Data for Monitoring Dataset Mentions in Research Papers [0.0]
  This paper presents a machine learning framework that automates dataset mention detection across research domains. We employ zero-shot extraction from research papers, an LLM-as-a-Judge for quality assessment, and a reasoning agent for refinement to generate a weakly supervised synthetic dataset. At inference, a ModernBERT-based classifier efficiently filters dataset mentions, reducing computational overhead while maintaining high recall.
  arXiv  Detail & Related papers  (2025-02-14T16:16:02Z)
• Data-driven tool wear prediction in milling, based on a process-integrated single-sensor approach [1.6574413179773764]
  This study explores data-driven methods, in particular deep learning, for tool wear prediction. The study evaluates several machine learning models, including convolutional neural networks (CNN), long short-term memory networks (LSTM), support vector machines (SVM) and decision trees. The ConvNeXt model has an exceptional performance, achieving a 99.1% accuracy in identifying tool wear using data from only four milling tools operated until they are worn.
  arXiv  Detail & Related papers  (2024-12-27T23:10:32Z)
• Data Advisor: Dynamic Data Curation for Safety Alignment of Large Language Models [79.65071553905021]
  We propose Data Advisor, a method for generating data that takes into account the characteristics of the desired dataset. Data Advisor monitors the status of the generated data, identifies weaknesses in the current dataset, and advises the next iteration of data generation.
  arXiv  Detail & Related papers  (2024-10-07T17:59:58Z)
• How to Train Data-Efficient LLMs [56.41105687693619]
  We study data-efficient approaches for pre-training language models (LLMs) We find that Ask-LLM and Density sampling are the best methods in their respective categories. In our comparison of 19 samplers, involving hundreds of evaluation tasks and pre-training runs, we find that Ask-LLM and Density are the best methods in their respective categories.
  arXiv  Detail & Related papers  (2024-02-15T02:27:57Z)
• LESS: Selecting Influential Data for Targeted Instruction Tuning [64.78894228923619]
  We propose LESS, an efficient algorithm to estimate data influences and perform Low-rank gradiEnt Similarity Search for instruction data selection. We show that training on a LESS-selected 5% of the data can often outperform training on the full dataset across diverse downstream tasks. Our method goes beyond surface form cues to identify data that the necessary reasoning skills for the intended downstream application.
  arXiv  Detail & Related papers  (2024-02-06T19:18:04Z)
• Dataset Factory: A Toolchain For Generative Computer Vision Datasets [0.9013233848500058]
  We propose a "dataset factory" that separates the storage and processing of samples from metadata. This enables data-centric operations at scale for machine learning teams and individual researchers.
  arXiv  Detail & Related papers  (2023-09-20T19:43:37Z)
• Demonstration of InsightPilot: An LLM-Empowered Automated Data Exploration System [48.62158108517576]
  We introduce InsightPilot, an automated data exploration system designed to simplify the data exploration process. InsightPilot automatically selects appropriate analysis intents, such as understanding, summarizing, and explaining. In brief, an IQuery is an abstraction and automation of data analysis operations, which mimics the approach of data analysts.
  arXiv  Detail & Related papers  (2023-04-02T07:27:49Z)
• Designing Data: Proactive Data Collection and Iteration for Machine Learning [12.295169687537395]
  Lack of diversity in data collection has caused significant failures in machine learning (ML) applications. New methods to track & manage data collection, iteration, and model training are necessary for evaluating whether datasets reflect real world variability.
  arXiv  Detail & Related papers  (2023-01-24T21:40:29Z)
• Fix your Models by Fixing your Datasets [0.6058427379240697]
  Current machine learning tools lack streamlined processes for improving the data quality. We introduce a systematic framework for finding noisy or mislabelled samples in the dataset. We demonstrate the efficacy of our framework on public as well as private enterprise datasets of two Fortune 500 companies.
  arXiv  Detail & Related papers  (2021-12-15T02:41:50Z)
• Automated Machine Learning Techniques for Data Streams [91.3755431537592]
  This paper surveys the state-of-the-art open-source AutoML tools, applies them to data collected from streams, and measures how their performance changes over time. The results show that off-the-shelf AutoML tools can provide satisfactory results but in the presence of concept drift, detection or adaptation techniques have to be applied to maintain the predictive accuracy over time.
  arXiv  Detail & Related papers  (2021-06-14T11:42:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.