Reproducibility of Machine Learning: Terminology, Recommendations and Open Issues

• URL: http://arxiv.org/abs/2302.12691v1
• Date: Fri, 24 Feb 2023 15:33:20 GMT
• Title: Reproducibility of Machine Learning: Terminology, Recommendations and Open Issues
• Authors: Riccardo Albertoni and Sara Colantonio and Piotr Skrzypczy\'nski and Jerzy Stefanowski
• Abstract summary: A crisis has been recently acknowledged by scientists and this seems to affect even more Artificial Intelligence and Machine Learning. We critically review the current literature on the topic and highlight the open issues. We identify key elements often overlooked in modern Machine Learning and provide novel recommendations for them.
• Score: 5.30596984761294
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reproducibility is one of the core dimensions that concur to deliver Trustworthy Artificial Intelligence. Broadly speaking, reproducibility can be defined as the possibility to reproduce the same or a similar experiment or method, thereby obtaining the same or similar results as the original scientists. It is an essential ingredient of the scientific method and crucial for gaining trust in relevant claims. A reproducibility crisis has been recently acknowledged by scientists and this seems to affect even more Artificial Intelligence and Machine Learning, due to the complexity of the models at the core of their recent successes. Notwithstanding the recent debate on Artificial Intelligence reproducibility, its practical implementation is still insufficient, also because many technical issues are overlooked. In this survey, we critically review the current literature on the topic and highlight the open issues. Our contribution is three-fold. We propose a concise terminological review of the terms coming into play. We collect and systematize existing recommendations for achieving reproducibility, putting forth the means to comply with them. We identify key elements often overlooked in modern Machine Learning and provide novel recommendations for them. We further specialize these for two critical application domains, namely the biomedical and physical artificial intelligence fields.

Related papers

• Automated Explanation Selection for Scientific Discovery [0.0]
  We propose a cycle of scientific discovery that combines machine learning with automated reasoning for the generation and the selection of explanations. We present a taxonomy of explanation selection problems that draws on insights from sociology and cognitive science.
  arXiv  Detail & Related papers  (2024-07-24T17:41:32Z)
• Leveraging large language models for nano synthesis mechanism explanation: solid foundations or mere conjectures? [12.874860522120326]
  We develop a benchmark consisting of 775 multiple-choice questions focusing on the mechanisms of gold nanoparticles synthesis. We propose a novel evaluation metric, the confidence-based score (c-score), which probes the output logits to derive the precise probability for the correct answer.
  arXiv  Detail & Related papers  (2024-07-12T02:05:59Z)
• Open-world Machine Learning: A Review and New Outlooks [83.6401132743407]
  This paper aims to provide a comprehensive introduction to the emerging open-world machine learning paradigm. It aims to help researchers build more powerful AI systems in their respective fields, and to promote the development of artificial general intelligence.
  arXiv  Detail & Related papers  (2024-03-04T06:25:26Z)
• A Review of Neuroscience-Inspired Machine Learning [58.72729525961739]
  Bio-plausible credit assignment is compatible with practically any learning condition and is energy-efficient. In this paper, we survey several vital algorithms that model bio-plausible rules of credit assignment in artificial neural networks. We conclude by discussing the future challenges that will need to be addressed in order to make such algorithms more useful in practical applications.
  arXiv  Detail & Related papers  (2024-02-16T18:05:09Z)
• A Neuro-mimetic Realization of the Common Model of Cognition via Hebbian Learning and Free Energy Minimization [55.11642177631929]
  Large neural generative models are capable of synthesizing semantically rich passages of text or producing complex images. We discuss the COGnitive Neural GENerative system, such an architecture that casts the Common Model of Cognition.
  arXiv  Detail & Related papers  (2023-10-14T23:28:48Z)
• Artificial Intelligence for Science in Quantum, Atomistic, and Continuum Systems [268.585904751315]
  New area of research known as AI for science (AI4Science) Areas aim at understanding the physical world from subatomic (wavefunctions and electron density), atomic (molecules, proteins, materials, and interactions), to macro (fluids, climate, and subsurface) scales. Key common challenge is how to capture physics first principles, especially symmetries, in natural systems by deep learning methods.
  arXiv  Detail & Related papers  (2023-07-17T12:14:14Z)
• The Age of Synthetic Realities: Challenges and Opportunities [85.058932103181]
  We highlight the crucial need for the development of forensic techniques capable of identifying harmful synthetic creations and distinguishing them from reality. Our focus extends to various forms of media, such as images, videos, audio, and text, as we examine how synthetic realities are crafted and explore approaches to detecting these malicious creations. This study is of paramount importance due to the rapid progress of AI generative techniques and their impact on the fundamental principles of Forensic Science.
  arXiv  Detail & Related papers  (2023-06-09T15:55:10Z)
• Building a Culture of Reproducibility in Academic Research [55.22219308265945]
  Reproducibility is an ideal that no researcher would dispute "in the abstract", but when aspirations meet the cold hard reality of the academic grind, often "loses out" In this essay, I share some personal experiences grappling with how to operationalize while balancing its demands against other priorities.
  arXiv  Detail & Related papers  (2022-12-27T16:03:50Z)
• When not to use machine learning: a perspective on potential and limitations [0.0]
  We highlight the guiding principles of data-driven modeling, how these principles imbue models with almost magical predictive power. We hope that the discussion to follow provides researchers throughout the sciences with a better understanding of when said techniques are appropriate.
  arXiv  Detail & Related papers  (2022-10-06T04:00:00Z)
• Reproducibility in machine learning for medical imaging [3.1390096961027076]
  This chapter intends at being an introduction to for researchers in the field of machine learning for medical imaging. For each of them, we aim at defining it, at describing the requirements to achieve it and at discussing its utility. The chapter ends with a discussion on the benefits of didactic and with a plea for a non-dogmatic approach to this concept and its implementation in research practice.
  arXiv  Detail & Related papers  (2022-09-12T09:00:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.