Reproducibility in Machine Learning-based Research: Overview, Barriers and Drivers

• URL: http://arxiv.org/abs/2406.14325v2
• Date: Tue, 2 Jul 2024 15:36:32 GMT
• Title: Reproducibility in Machine Learning-based Research: Overview, Barriers and Drivers
• Authors: Harald Semmelrock, Tony Ross-Hellauer, Simone Kopeinik, Dieter Theiler, Armin Haberl, Stefan Thalmann, Dominik Kowald,
• Abstract summary: Research in various fields is currently experiencing challenges regarding awareness of results. This problem is also prevalent in machine learning (ML) research. The level of in ML-driven research remains unsatisfactory.
• Score: 1.4841630983274845
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Research in various fields is currently experiencing challenges regarding the reproducibility of results. This problem is also prevalent in machine learning (ML) research. The issue arises, for example, due to unpublished data and/or source code and the sensitivity of ML training conditions. Although different solutions have been proposed to address this issue, such as using ML platforms, the level of reproducibility in ML-driven research remains unsatisfactory. Therefore, in this article, we discuss the reproducibility of ML-driven research with three main aims: (i) identifying the barriers to reproducibility when applying ML in research as well as categorize the barriers to different types of reproducibility (description, code, data, and experiment reproducibility), (ii) discussing potential drivers such as tools, practices, and interventions that support ML reproducibility, as well as distinguish between technology-driven drivers, procedural drivers, and drivers related to awareness and education, and (iii) mapping the drivers to the barriers. With this work, we hope to provide insights and to contribute to the decision-making process regarding the adoption of different solutions to support ML reproducibility.

Related papers

• Maintainability Challenges in ML: A Systematic Literature Review [5.669063174637433]
  This study aims to identify and synthesise the maintainability challenges in different stages of the Machine Learning workflow. We screened more than 13000 papers, then selected and qualitatively analysed 56 of them.
  arXiv  Detail & Related papers  (2024-08-17T13:24:15Z)
• Retrieval-Enhanced Machine Learning: Synthesis and Opportunities [60.34182805429511]
  Retrieval-enhancement can be extended to a broader spectrum of machine learning (ML) This work introduces a formal framework of this paradigm, Retrieval-Enhanced Machine Learning (REML), by synthesizing the literature in various domains in ML with consistent notations which is missing from the current literature. The goal of this work is to equip researchers across various disciplines with a comprehensive, formally structured framework of retrieval-enhanced models, thereby fostering interdisciplinary future research.
  arXiv  Detail & Related papers  (2024-07-17T20:01:21Z)
• Reproducibility in Machine Learning-Driven Research [1.7936835766396748]
  Research is facing a viability crisis, in which the results and findings of many studies are difficult or even impossible to reproduce. This is also the case in machine learning (ML) and artificial intelligence (AI) research. Although different solutions to address this issue are discussed in the research community such as using ML platforms, the level of in ML-driven research is not increasing substantially.
  arXiv  Detail & Related papers  (2023-07-19T07:00:22Z)
• A Review of Machine Learning Methods Applied to Structural Dynamics and Vibroacoustic [0.0]
  Three main applications in Vibroacoustic (SD&V) have taken advantage of Machine Learning (ML) In Structural Health Monitoring, ML detection and prognosis lead to safe operation and optimized maintenance schedules. System identification and control design are leveraged by ML techniques in Active Noise Control and Active Vibration Control. The so-called ML-based surrogate models provide fast alternatives to costly simulations, enabling robust and optimized product design.
  arXiv  Detail & Related papers  (2022-04-13T13:16:21Z)
• The challenge of reproducible ML: an empirical study on the impact of bugs [6.862925771672299]
  In this paper, we establish the fundamental factors that cause non-determinism in Machine Learning systems. A framework, ReproduceML, is then introduced for deterministic evaluation of ML experiments in a real, controlled environment. This study attempts to quantify the impact that the occurrence of bugs in a popular ML framework, PyTorch, has on the performance of trained models.
  arXiv  Detail & Related papers  (2021-09-09T01:36:39Z)
• Panoramic Learning with A Standardized Machine Learning Formalism [116.34627789412102]
  This paper presents a standardized equation of the learning objective, that offers a unifying understanding of diverse ML algorithms. It also provides guidance for mechanic design of new ML solutions, and serves as a promising vehicle towards panoramic learning with all experiences.
  arXiv  Detail & Related papers  (2021-08-17T17:44:38Z)
• Practical Machine Learning Safety: A Survey and Primer [81.73857913779534]
  Open-world deployment of Machine Learning algorithms in safety-critical applications such as autonomous vehicles needs to address a variety of ML vulnerabilities. New models and training techniques to reduce generalization error, achieve domain adaptation, and detect outlier examples and adversarial attacks. Our organization maps state-of-the-art ML techniques to safety strategies in order to enhance the dependability of the ML algorithm from different aspects.
  arXiv  Detail & Related papers  (2021-06-09T05:56:42Z)
• Inspect, Understand, Overcome: A Survey of Practical Methods for AI Safety [54.478842696269304]
  The use of deep neural networks (DNNs) in safety-critical applications is challenging due to numerous model-inherent shortcomings. In recent years, a zoo of state-of-the-art techniques aiming to address these safety concerns has emerged. Our paper addresses both machine learning experts and safety engineers.
  arXiv  Detail & Related papers  (2021-04-29T09:54:54Z)
• Understanding the Usability Challenges of Machine Learning In High-Stakes Decision Making [67.72855777115772]
  Machine learning (ML) is being applied to a diverse and ever-growing set of domains. In many cases, domain experts -- who often have no expertise in ML or data science -- are asked to use ML predictions to make high-stakes decisions. We investigate the ML usability challenges present in the domain of child welfare screening through a series of collaborations with child welfare screeners.
  arXiv  Detail & Related papers  (2021-03-02T22:50:45Z)
• Machine Learning Towards Intelligent Systems: Applications, Challenges, and Opportunities [8.68311678910946]
  Machine learning (ML) provides a mechanism for humans to process large amounts of data. This review focuses on some of the fields and applications such as education, healthcare, network security, banking and finance, and social media.
  arXiv  Detail & Related papers  (2021-01-11T01:32:15Z)
• Machine Learning in Nano-Scale Biomedical Engineering [77.75587007080894]
  We review the existing research regarding the use of machine learning in nano-scale biomedical engineering. The main challenges that can be formulated as ML problems are classified into the three main categories. For each of the presented methodologies, special emphasis is given to its principles, applications, and limitations.
  arXiv  Detail & Related papers  (2020-08-05T15:45:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.