KGLiDS: A Platform for Semantic Abstraction, Linking, and Automation of Data Science

• URL: http://arxiv.org/abs/2303.02204v4
• Date: Wed, 12 Jun 2024 07:26:35 GMT
• Title: KGLiDS: A Platform for Semantic Abstraction, Linking, and Automation of Data Science
• Authors: Mossad Helali, Niki Monjazeb, Shubham Vashisth, Philippe Carrier, Ahmed Helal, Antonio Cavalcante, Khaled Ammar, Katja Hose, Essam Mansour,
• Abstract summary: This paper presents a scalable platform, KGLiDS, that employs machine learning and knowledge graph technologies to abstract and capture the semantics of data science artifacts and their connections. Based on this information, KGLiDS enables various downstream applications, such as data discovery and pipeline automation.
• Score: 4.120803087965204
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In recent years, we have witnessed the growing interest from academia and industry in applying data science technologies to analyze large amounts of data. In this process, a myriad of artifacts (datasets, pipeline scripts, etc.) are created. However, there has been no systematic attempt to holistically collect and exploit all the knowledge and experiences that are implicitly contained in those artifacts. Instead, data scientists recover information and expertise from colleagues or learn via trial and error. Hence, this paper presents a scalable platform, KGLiDS, that employs machine learning and knowledge graph technologies to abstract and capture the semantics of data science artifacts and their connections. Based on this information, KGLiDS enables various downstream applications, such as data discovery and pipeline automation. Our comprehensive evaluation covers use cases in data discovery, data cleaning, transformation, and AutoML. It shows that KGLiDS is significantly faster with a lower memory footprint than the state-of-the-art systems while achieving comparable or better accuracy.

Related papers

• Web-Scale Visual Entity Recognition: An LLM-Driven Data Approach [56.55633052479446]
  Web-scale visual entity recognition presents significant challenges due to the lack of clean, large-scale training data. We propose a novel methodology to curate such a dataset, leveraging a multimodal large language model (LLM) for label verification, metadata generation, and rationale explanation. Experiments demonstrate that models trained on this automatically curated data achieve state-of-the-art performance on web-scale visual entity recognition tasks.
  arXiv  Detail & Related papers  (2024-10-31T06:55:24Z)
• DSBench: How Far Are Data Science Agents to Becoming Data Science Experts? [58.330879414174476]
  We introduce DSBench, a benchmark designed to evaluate data science agents with realistic tasks. This benchmark includes 466 data analysis tasks and 74 data modeling tasks, sourced from Eloquence and Kaggle competitions. Our evaluation of state-of-the-art LLMs, LVLMs, and agents shows that they struggle with most tasks, with the best agent solving only 34.12% of data analysis tasks and achieving a 34.74% Relative Performance Gap (RPG)
  arXiv  Detail & Related papers  (2024-09-12T02:08:00Z)
• How to Do Machine Learning with Small Data? -- A Review from an Industrial Perspective [1.443696537295348]
  Authors focus on interpreting the general term of "small data" and their engineering and industrial application role. Small data is defined in terms of various characteristics compared to big data, and a machine learning formalism was introduced. Five critical challenges of machine learning with small data in industrial applications are presented.
  arXiv  Detail & Related papers  (2023-11-13T07:39:13Z)
• Position Paper on Dataset Engineering to Accelerate Science [1.952708415083428]
  In this work, we will use the token ittextdataset to designate a structured set of data built to perform a well-defined task. Specifically, in science, each area has unique forms to organize, gather and handle its datasets. We advocate that science and engineering discovery processes are extreme instances of the need for such organization on datasets.
  arXiv  Detail & Related papers  (2023-03-09T19:07:40Z)
• A Vision for Semantically Enriched Data Science [19.604667287258724]
  Key areas such as utilizing domain knowledge and data semantics are areas where we have seen little automation. We envision how leveraging "semantic" understanding and reasoning on data in combination with novel tools for data science automation can help with consistent and explainable data augmentation and transformation.
  arXiv  Detail & Related papers  (2023-03-02T16:03:12Z)
• TRoVE: Transforming Road Scene Datasets into Photorealistic Virtual Environments [84.6017003787244]
  This work proposes a synthetic data generation pipeline to address the difficulties and domain-gaps present in simulated datasets. We show that using annotations and visual cues from existing datasets, we can facilitate automated multi-modal data generation.
  arXiv  Detail & Related papers  (2022-08-16T20:46:08Z)
• Knowledge Representation in Digital Agriculture: A Step Towards Standardised Model [4.286327408435937]
  We propose a novel-based knowledge map to represent and store data mining in crop farming. The proposed model consists of six main sets: concepts, attributes, relations, transformations, instances, and states. This paper also proposes an architecture for handling this knowledge-based model.
  arXiv  Detail & Related papers  (2022-07-15T20:31:56Z)
• SOLIS -- The MLOps journey from data acquisition to actionable insights [62.997667081978825]
  In this paper we present a unified deployment pipeline and freedom-to-operate approach that supports all requirements while using basic cross-platform tensor framework and script language engines. This approach however does not supply the needed procedures and pipelines for the actual deployment of machine learning capabilities in real production grade systems.
  arXiv  Detail & Related papers  (2021-12-22T14:45:37Z)
• Synthetic Data: Opening the data floodgates to enable faster, more directed development of machine learning methods [96.92041573661407]
  Many ground-breaking advancements in machine learning can be attributed to the availability of a large volume of rich data. Many large-scale datasets are highly sensitive, such as healthcare data, and are not widely available to the machine learning community. Generating synthetic data with privacy guarantees provides one such solution.
  arXiv  Detail & Related papers  (2020-12-08T17:26:10Z)
• Scaling Systematic Literature Reviews with Machine Learning Pipelines [57.82662094602138]
  Systematic reviews entail the extraction of data from scientific documents. We construct a pipeline that automates each of these aspects, and experiment with many human-time vs. system quality trade-offs. We find that we can get surprising accuracy and generalisability of the whole pipeline system with only 2 weeks of human-expert annotation.
  arXiv  Detail & Related papers  (2020-10-09T16:19:42Z)
• Data science on industrial data -- Today's challenges in brown field applications [0.0]
  This paper shows state of the art and what to expect when working with stock machines in the field. A major focus in this paper is on data collection which can be more cumbersome than most people might expect. Data quality for machine learning applications is a challenge once leaving the laboratory.
  arXiv  Detail & Related papers  (2020-06-10T10:05:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.