Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program)

• URL: http://arxiv.org/abs/2003.12206v4
• Date: Wed, 30 Dec 2020 21:32:34 GMT
• Title: Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program)
• Authors: Joelle Pineau, Philippe Vincent-Lamarre, Koustuv Sinha, Vincent Larivi\`ere, Alina Beygelzimer, Florence d'Alch\'e-Buc, Emily Fox, Hugo Larochelle
• Abstract summary: Reproducibility is obtaining similar results as presented in a paper or talk, using the same code and data (when available) In 2019, the Neural Information Processing Systems (NeurIPS) conference introduced a program, designed to improve the standards across the community for how we conduct, communicate, and evaluate machine learning research. In this paper, we describe each of these components, how they were deployed, as well as what we were able to learn from this initiative.
• Score: 43.55295847227261
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: One of the challenges in machine learning research is to ensure that presented and published results are sound and reliable. Reproducibility, that is obtaining similar results as presented in a paper or talk, using the same code and data (when available), is a necessary step to verify the reliability of research findings. Reproducibility is also an important step to promote open and accessible research, thereby allowing the scientific community to quickly integrate new findings and convert ideas to practice. Reproducibility also promotes the use of robust experimental workflows, which potentially reduce unintentional errors. In 2019, the Neural Information Processing Systems (NeurIPS) conference, the premier international conference for research in machine learning, introduced a reproducibility program, designed to improve the standards across the community for how we conduct, communicate, and evaluate machine learning research. The program contained three components: a code submission policy, a community-wide reproducibility challenge, and the inclusion of the Machine Learning Reproducibility checklist as part of the paper submission process. In this paper, we describe each of these components, how it was deployed, as well as what we were able to learn from this initiative.

Related papers

• Retrieval-Enhanced Machine Learning: Synthesis and Opportunities [60.34182805429511]
  Retrieval-enhancement can be extended to a broader spectrum of machine learning (ML) This work introduces a formal framework of this paradigm, Retrieval-Enhanced Machine Learning (REML), by synthesizing the literature in various domains in ML with consistent notations which is missing from the current literature. The goal of this work is to equip researchers across various disciplines with a comprehensive, formally structured framework of retrieval-enhanced models, thereby fostering interdisciplinary future research.
  arXiv  Detail & Related papers  (2024-07-17T20:01:21Z)
• Application-Driven Innovation in Machine Learning [56.85396167616353]
  We describe the paradigm of application-driven research in machine learning. We show how this approach can productively synergize with methods-driven work. Despite these benefits, we find that reviewing, hiring, and teaching practices in machine learning often hold back application-driven innovation.
  arXiv  Detail & Related papers  (2024-03-26T04:59:27Z)
• Reproducibility and Geometric Intrinsic Dimensionality: An Investigation on Graph Neural Network Research [0.0]
  Building on these efforts we turn towards another critical challenge in machine learning, namely the curse of dimensionality. Using the closely linked concept of intrinsic dimension we investigate to which the used machine learning models are influenced by the extend dimension of the data sets they are trained on.
  arXiv  Detail & Related papers  (2024-03-13T11:44:30Z)
• A pragmatic workflow for research software engineering in computational science [0.0]
  University research groups in Computational Science and Engineering (CSE) generally lack dedicated funding and personnel for Research Software Engineering (RSE) RSE shifts the focus away from sustainable research software development and reproducible results. We propose a RSE workflow for CSE that addresses these challenges, that improves the quality of research output in CSE.
  arXiv  Detail & Related papers  (2023-10-02T08:04:12Z)
• Benchopt: Reproducible, efficient and collaborative optimization benchmarks [67.29240500171532]
  Benchopt is a framework to automate, reproduce and publish optimization benchmarks in machine learning. Benchopt simplifies benchmarking for the community by providing an off-the-shelf tool for running, sharing and extending experiments.
  arXiv  Detail & Related papers  (2022-06-27T16:19:24Z)
• Retrieval-Enhanced Machine Learning [110.5237983180089]
  We describe a generic retrieval-enhanced machine learning framework, which includes a number of existing models as special cases. REML challenges information retrieval conventions, presenting opportunities for novel advances in core areas, including optimization. REML research agenda lays a foundation for a new style of information access research and paves a path towards advancing machine learning and artificial intelligence.
  arXiv  Detail & Related papers  (2022-05-02T21:42:45Z)
• Machine Learning State-of-the-Art with Uncertainties [3.4123736336071864]
  We conduct an exemplary image classification study in order to demonstrate how confidence intervals around accuracy measurements can greatly enhance the communication of research results. We make suggestions for improving the authoring and reviewing process of machine learning articles.
  arXiv  Detail & Related papers  (2022-04-11T15:06:26Z)
• DeepZensols: Deep Natural Language Processing Framework [23.56171046067646]
  This work is a framework that is able to reproduce consistent results. It provides a means of easily creating, training, and evaluating natural language processing (NLP) deep learning (DL) models.
  arXiv  Detail & Related papers  (2021-09-08T01:16:05Z)
• BUSTLE: Bottom-Up Program Synthesis Through Learning-Guided Exploration [72.88493072196094]
  We present a new synthesis approach that leverages learning to guide a bottom-up search over programs. In particular, we train a model to prioritize compositions of intermediate values during search conditioned on a set of input-output examples. We show that the combination of learning and bottom-up search is remarkably effective, even with simple supervised learning approaches.
  arXiv  Detail & Related papers  (2020-07-28T17:46:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.