Machine Learning Pipelines: Provenance, Reproducibility and FAIR Data Principles

• URL: http://arxiv.org/abs/2006.12117v1
• Date: Mon, 22 Jun 2020 10:17:34 GMT
• Title: Machine Learning Pipelines: Provenance, Reproducibility and FAIR Data Principles
• Authors: Sheeba Samuel, Frank L\"offler, Birgitta K\"onig-Ries
• Abstract summary: We describe our goals and initial steps in supporting the end-to-end of machine learning pipelines. We investigate which factors beyond the availability of source code and datasets influence the influence of ML experiments. We propose ways to apply FAIR data practices to ML experiments.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Machine learning (ML) is an increasingly important scientific tool supporting decision making and knowledge generation in numerous fields. With this, it also becomes more and more important that the results of ML experiments are reproducible. Unfortunately, that often is not the case. Rather, ML, similar to many other disciplines, faces a reproducibility crisis. In this paper, we describe our goals and initial steps in supporting the end-to-end reproducibility of ML pipelines. We investigate which factors beyond the availability of source code and datasets influence reproducibility of ML experiments. We propose ways to apply FAIR data practices to ML workflows. We present our preliminary results on the role of our tool, ProvBook, in capturing and comparing provenance of ML experiments and their reproducibility using Jupyter Notebooks.

Related papers

• Recent Advances on Machine Learning for Computational Fluid Dynamics: A Survey [51.87875066383221]
  This paper introduces fundamental concepts, traditional methods, and benchmark datasets, then examine the various roles Machine Learning plays in improving CFD. We highlight real-world applications of ML for CFD in critical scientific and engineering disciplines, including aerodynamics, combustion, atmosphere & ocean science, biology fluid, plasma, symbolic regression, and reduced order modeling. We draw the conclusion that ML is poised to significantly transform CFD research by enhancing simulation accuracy, reducing computational time, and enabling more complex analyses of fluid dynamics.
  arXiv  Detail & Related papers  (2024-08-22T07:33:11Z)
• MLXP: A Framework for Conducting Replicable Experiments in Python [63.37350735954699]
  We propose MLXP, an open-source, simple, and lightweight experiment management tool based on Python. It streamlines the experimental process with minimal overhead while ensuring a high level of practitioner overhead.
  arXiv  Detail & Related papers  (2024-02-21T14:22:20Z)
• DataDreamer: A Tool for Synthetic Data Generation and Reproducible LLM Workflows [72.40917624485822]
  We introduce DataDreamer, an open source Python library that allows researchers to implement powerful large language models. DataDreamer also helps researchers adhere to best practices that we propose to encourage open science.
  arXiv  Detail & Related papers  (2024-02-16T00:10:26Z)
• Julearn: an easy-to-use library for leakage-free evaluation and inspection of ML models [0.23301643766310373]
  We present the rationale behind julearn's design, its core features, and showcase three examples of previously-published research projects. Julearn aims to simplify the entry into the machine learning world by providing an easy-to-use environment with built in guards against some of the most common ML pitfalls.
  arXiv  Detail & Related papers  (2023-10-19T08:21:12Z)
• Human-in-the-loop: The future of Machine Learning in Automated Electron Microscopy [0.6760163180787716]
  We discuss some considerations in designing ML-based active experiments. The likely strategy for the next several years will be human-in-the-loop automated experiments.
  arXiv  Detail & Related papers  (2023-10-08T05:26:32Z)
• From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuning [52.257422715393574]
  We introduce a self-guided methodology for Large Language Models (LLMs) to autonomously discern and select cherry samples from open-source datasets. Our key innovation, the Instruction-Following Difficulty (IFD) metric, emerges as a pivotal metric to identify discrepancies between a model's expected responses and its intrinsic generation capability.
  arXiv  Detail & Related papers  (2023-08-23T09:45:29Z)
• Closing the loop: Autonomous experiments enabled by machine-learning-based online data analysis in synchrotron beamline environments [80.49514665620008]
  Machine learning can be used to enhance research involving large or rapidly generated datasets. In this study, we describe the incorporation of ML into a closed-loop workflow for X-ray reflectometry (XRR) We present solutions that provide an elementary data analysis in real time during the experiment without introducing the additional software dependencies in the beamline control software environment.
  arXiv  Detail & Related papers  (2023-06-20T21:21:19Z)
• Reasonable Scale Machine Learning with Open-Source Metaflow [2.637746074346334]
  We argue that re-purposing existing tools won't solve the current productivity issues. We introduce Metaflow, an open-source framework for ML projects explicitly designed to boost the productivity of data practitioners.
  arXiv  Detail & Related papers  (2023-03-21T11:28:09Z)
• The challenge of reproducible ML: an empirical study on the impact of bugs [6.862925771672299]
  In this paper, we establish the fundamental factors that cause non-determinism in Machine Learning systems. A framework, ReproduceML, is then introduced for deterministic evaluation of ML experiments in a real, controlled environment. This study attempts to quantify the impact that the occurrence of bugs in a popular ML framework, PyTorch, has on the performance of trained models.
  arXiv  Detail & Related papers  (2021-09-09T01:36:39Z)
• Understanding the Usability Challenges of Machine Learning In High-Stakes Decision Making [67.72855777115772]
  Machine learning (ML) is being applied to a diverse and ever-growing set of domains. In many cases, domain experts -- who often have no expertise in ML or data science -- are asked to use ML predictions to make high-stakes decisions. We investigate the ML usability challenges present in the domain of child welfare screening through a series of collaborations with child welfare screeners.
  arXiv  Detail & Related papers  (2021-03-02T22:50:45Z)
• Insights into Performance Fitness and Error Metrics for Machine Learning [1.827510863075184]
  Machine learning (ML) is the field of training machines to achieve high level of cognition and perform human-like analysis. This paper examines a number of the most commonly-used performance fitness and error metrics for regression and classification algorithms.
  arXiv  Detail & Related papers  (2020-05-17T22:59:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.