What Do Machine Learning Researchers Mean by "Reproducible"?

• URL: http://arxiv.org/abs/2412.03854v1
• Date: Thu, 05 Dec 2024 04:04:39 GMT
• Title: What Do Machine Learning Researchers Mean by "Reproducible"?
• Authors: Edward Raff, Michel Benaroch, Sagar Samtani, Andrew L. Farris,
• Abstract summary: We try to clarify the scope of "reproducibility" as displayed by the community at large. We see that each of these areas contains many works that do not advertise themselves as being about "reproducibility"
• Score: 47.893726475815434
• License:
• Abstract: The concern that Artificial Intelligence (AI) and Machine Learning (ML) are entering a "reproducibility crisis" has spurred significant research in the past few years. Yet with each paper, it is often unclear what someone means by "reproducibility". Our work attempts to clarify the scope of "reproducibility" as displayed by the community at large. In doing so, we propose to refine the research to eight general topic areas. In this light, we see that each of these areas contains many works that do not advertise themselves as being about "reproducibility", in part because they go back decades before the matter came to broader attention.

Related papers

• A Survey on Game Playing Agents and Large Models: Methods, Applications, and Challenges [29.74898680986507]
  We review the current landscape of LM usage in regards to complex game playing scenarios and the challenges still open. We present our perspective on promising future research avenues for the advancement of LMs in games.
  arXiv  Detail & Related papers  (2024-03-15T12:37:12Z)
• Open-world Machine Learning: A Review and New Outlooks [83.6401132743407]
  This paper aims to provide a comprehensive introduction to the emerging open-world machine learning paradigm. It aims to help researchers build more powerful AI systems in their respective fields, and to promote the development of artificial general intelligence.
  arXiv  Detail & Related papers  (2024-03-04T06:25:26Z)
• Brain in a Vat: On Missing Pieces Towards Artificial General Intelligence in Large Language Models [83.63242931107638]
  We propose four characteristics of generally intelligent agents. We argue that active engagement with objects in the real world delivers more robust signals for forming conceptual representations. We conclude by outlining promising future research directions in the field of artificial general intelligence.
  arXiv  Detail & Related papers  (2023-07-07T13:58:16Z)
• Reproducibility of Machine Learning: Terminology, Recommendations and Open Issues [5.30596984761294]
  A crisis has been recently acknowledged by scientists and this seems to affect even more Artificial Intelligence and Machine Learning. We critically review the current literature on the topic and highlight the open issues. We identify key elements often overlooked in modern Machine Learning and provide novel recommendations for them.
  arXiv  Detail & Related papers  (2023-02-24T15:33:20Z)
• Building a Culture of Reproducibility in Academic Research [55.22219308265945]
  Reproducibility is an ideal that no researcher would dispute "in the abstract", but when aspirations meet the cold hard reality of the academic grind, often "loses out" In this essay, I share some personal experiences grappling with how to operationalize while balancing its demands against other priorities.
  arXiv  Detail & Related papers  (2022-12-27T16:03:50Z)
• Reproducibility in machine learning for medical imaging [3.1390096961027076]
  This chapter intends at being an introduction to for researchers in the field of machine learning for medical imaging. For each of them, we aim at defining it, at describing the requirements to achieve it and at discussing its utility. The chapter ends with a discussion on the benefits of didactic and with a plea for a non-dogmatic approach to this concept and its implementation in research practice.
  arXiv  Detail & Related papers  (2022-09-12T09:00:04Z)
• From Psychological Curiosity to Artificial Curiosity: Curiosity-Driven Learning in Artificial Intelligence Tasks [56.20123080771364]
  Psychological curiosity plays a significant role in human intelligence to enhance learning through exploration and information acquisition. In the Artificial Intelligence (AI) community, artificial curiosity provides a natural intrinsic motivation for efficient learning. CDL has become increasingly popular, where agents are self-motivated to learn novel knowledge.
  arXiv  Detail & Related papers  (2022-01-20T17:07:03Z)
• Semantics for Robotic Mapping, Perception and Interaction: A Survey [93.93587844202534]
  Study of understanding dictates what does the world "mean" to a robot. With humans and robots increasingly operating in the same world, the prospects of human-robot interaction also bring semantics into the picture. Driven by need, as well as by enablers like increasing availability of training data and computational resources, semantics is a rapidly growing research area in robotics.
  arXiv  Detail & Related papers  (2021-01-02T12:34:39Z)
• Research Reproducibility as a Survival Analysis [22.66983713481359]
  We consider modeling a paper as a survival analysis problem. We show how a survival analysis allows us to draw new insights that better explain prior longitudinal data.
  arXiv  Detail & Related papers  (2020-12-17T20:56:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.