Data Pipeline Quality: Influencing Factors, Root Causes of Data-related Issues, and Processing Problem Areas for Developers

• URL: http://arxiv.org/abs/2309.07067v1
• Date: Wed, 13 Sep 2023 16:28:10 GMT
• Title: Data Pipeline Quality: Influencing Factors, Root Causes of Data-related Issues, and Processing Problem Areas for Developers
• Abstract summary: This article first introduces a taxonomy of 41 factors that influence the ability of data pipelines to provide quality data. Data, infrastructure, life cycle management, development & deployment, and processing were found to be the main influencing themes.
• Score: 4.473327661758546
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Data pipelines are an integral part of various modern data-driven systems. However, despite their importance, they are often unreliable and deliver poor-quality data. A critical step toward improving this situation is a solid understanding of the aspects contributing to the quality of data pipelines. Therefore, this article first introduces a taxonomy of 41 factors that influence the ability of data pipelines to provide quality data. The taxonomy is based on a multivocal literature review and validated by eight interviews with experts from the data engineering domain. Data, infrastructure, life cycle management, development & deployment, and processing were found to be the main influencing themes. Second, we investigate the root causes of data-related issues, their location in data pipelines, and the main topics of data pipeline processing issues for developers by mining GitHub projects and Stack Overflow posts. We found data-related issues to be primarily caused by incorrect data types (33%), mainly occurring in the data cleaning stage of pipelines (35%). Data integration and ingestion tasks were found to be the most asked topics of developers, accounting for nearly half (47%) of all questions. Compatibility issues were found to be a separate problem area in addition to issues corresponding to the usual data pipeline processing areas (i.e., data loading, ingestion, integration, cleaning, and transformation). These findings suggest that future research efforts should focus on analyzing compatibility and data type issues in more depth and assisting developers in data integration and ingestion tasks. The proposed taxonomy is valuable to practitioners in the context of quality assurance activities and fosters future research into data pipeline quality.

Related papers

• LAND: A Longitudinal Analysis of Neuromorphic Datasets [47.84658158364453]
  This review captures the existing neuromorphic datasets, covering over 423 datasets.<n>Analysing these datasets shows the difficulties arising from their size, the lack of standardisation, and difficulties in accessing the actual data.<n>A more important concern is the rise of synthetic datasets, created by either simulation or video-to-events methods.
  arXiv  Detail & Related papers  (2026-02-17T19:46:33Z)
• Analyzing GitHub Issues and Pull Requests in nf-core Pipelines: Insights into nf-core Pipeline Repositories [4.902956965439]
  Nextflow's nf-core community curates standardized, peer-reviewed pipelines that follow strict testing, documentation, and governance guidelines.<n>This paper presents an empirical study of 25,173 issues and pull requests from these pipelines to uncover recurring challenges, management practices, and perceived difficulties.<n>We identify 13 key challenges, including pipeline development and integration, bug fixing, integrating genomic data, managing CI configurations, and handling version updates.
  arXiv  Detail & Related papers  (2026-01-14T16:34:00Z)
• OpenDataArena: A Fair and Open Arena for Benchmarking Post-Training Dataset Value [74.80873109856563]
  OpenDataArena (ODA) is a holistic and open platform designed to benchmark the intrinsic value of post-training data.<n>ODA establishes a comprehensive ecosystem comprising four key pillars: (i) a unified training-evaluation pipeline that ensures fair, open comparisons across diverse models; (ii) a multi-dimensional scoring framework that profiles data quality along tens of distinct axes; and (iii) an interactive data lineage explorer to visualize dataset genealogy and dissect component sources.
  arXiv  Detail & Related papers  (2025-12-16T03:33:24Z)
• Benchmarking Deep Search over Heterogeneous Enterprise Data [73.55304268238474]
  We present a new benchmark for evaluating a form of retrieval-augmented generation (RAG)<n>RAG requires source-aware, multi-hop reasoning over diverse, sparsed, but related sources.<n>We build it using a synthetic data pipeline that simulates business across product planning, development, and support stages.
  arXiv  Detail & Related papers  (2025-06-29T08:34:59Z)
• IssueCourier: Multi-Relational Heterogeneous Temporal Graph Neural Network for Open-Source Issue Assignment [5.1987901165589]
  Issue assignment plays a critical role in open-source software (OSS) maintenance.<n>We propose IssueCourier, a novel Multi-Relational Heterogeneous Temporal Graph Neural Network approach for issue assignment.<n>We show that IssueCourier can improve over the best baseline up to 45.49% in top-1 and 31.97% in MRR.
  arXiv  Detail & Related papers  (2025-05-16T13:03:26Z)
• Towards Understanding the Impact of Data Bugs on Deep Learning Models in Software Engineering [13.17302533571231]
  Deep learning (DL) systems are prone to bugs from many sources, including training data. Existing literature suggests that bugs in training data are highly prevalent. We investigate three types of data prevalent in software engineering tasks: code-based, text-based, and metric-based.
  arXiv  Detail & Related papers  (2024-11-19T00:28:20Z)
• Enhancing Data Quality in Federated Fine-Tuning of Foundation Models [54.757324343062734]
  We propose a data quality control pipeline for federated fine-tuning of foundation models. This pipeline computes scores reflecting the quality of training data and determines a global threshold for a unified standard. Our experiments show that the proposed quality control pipeline facilitates the effectiveness and reliability of the model training, leading to better performance.
  arXiv  Detail & Related papers  (2024-03-07T14:28:04Z)
• Investigating the Quality of DermaMNIST and Fitzpatrick17k Dermatological Image Datasets [19.128392861461297]
  We conduct meticulous analyses of two popular dermatological image datasets: DermaMNIST and Fitzpatrick17k. We uncover data quality issues, measure the effects of these problems on the benchmark results, and propose corrections to the datasets.
  arXiv  Detail & Related papers  (2024-01-25T20:29:01Z)
• A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicity [84.6421260559093]
  This study is the largest set of experiments to validate, quantify, and expose undocumented intuitions about text pretraining. Our findings indicate there does not exist a one-size-fits-all solution to filtering training data.
  arXiv  Detail & Related papers  (2023-05-22T15:57:53Z)
• A Comprehensive Survey of Dataset Distillation [73.15482472726555]
  It has become challenging to handle the unlimited growth of data with limited computing power. Deep learning technology has developed unprecedentedly in the last decade. This paper provides a holistic understanding of dataset distillation from multiple aspects.
  arXiv  Detail & Related papers  (2023-01-13T15:11:38Z)
• Advanced Data Augmentation Approaches: A Comprehensive Survey and Future directions [57.30984060215482]
  We provide a background of data augmentation, a novel and comprehensive taxonomy of reviewed data augmentation techniques, and the strengths and weaknesses (wherever possible) of each technique. We also provide comprehensive results of the data augmentation effect on three popular computer vision tasks, such as image classification, object detection and semantic segmentation.
  arXiv  Detail & Related papers  (2023-01-07T11:37:32Z)
• A Survey of Dataset Refinement for Problems in Computer Vision Datasets [11.45536223418548]
  Large-scale datasets have played a crucial role in the advancement of computer vision. They often suffer from problems such as class imbalance, noisy labels, dataset bias, or high resource costs. Various data-centric solutions have been proposed to solve the dataset problems. They improve the quality of datasets by re-organizing them, which we call dataset refinement.
  arXiv  Detail & Related papers  (2022-10-21T03:58:43Z)
• TRoVE: Transforming Road Scene Datasets into Photorealistic Virtual Environments [84.6017003787244]
  This work proposes a synthetic data generation pipeline to address the difficulties and domain-gaps present in simulated datasets. We show that using annotations and visual cues from existing datasets, we can facilitate automated multi-modal data generation.
  arXiv  Detail & Related papers  (2022-08-16T20:46:08Z)
• Augment & Valuate : A Data Enhancement Pipeline for Data-Centric AI [19.358073575300004]
  We propose a data-centric approach to address the fundamental distributional and semantic properties of dataset with black box models. We achieve 84.711% test accuracy (ranked #6, Honorable Mention in the Most Innovative) in the Data-Centric AI competition only with the provided dataset.
  arXiv  Detail & Related papers  (2021-12-07T17:22:44Z)
• Competency Problems: On Finding and Removing Artifacts in Language Data [50.09608320112584]
  We argue that for complex language understanding tasks, all simple feature correlations are spurious. We theoretically analyze the difficulty of creating data for competency problems when human bias is taken into account.
  arXiv  Detail & Related papers  (2021-04-17T21:34:10Z)
• Challenges in Benchmarking Stream Learning Algorithms with Real-world Data [2.861782696432711]
  Streaming data are increasingly present in real-world applications such as sensor measurements, satellite data feed, stock market, and financial data. The data stream mining community still faces some primary challenges and difficulties related to the comparison and evaluation of new proposals. We propose a new public data repository for benchmarking stream algorithms with real-world data.
  arXiv  Detail & Related papers  (2020-04-30T21:31:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.