Reproducibility, Replicability, and Repeatability: A survey of
reproducible research with a focus on high performance computing
- URL: http://arxiv.org/abs/2402.07530v1
- Date: Mon, 12 Feb 2024 09:59:11 GMT
- Title: Reproducibility, Replicability, and Repeatability: A survey of
reproducible research with a focus on high performance computing
- Authors: Benjamin A. Antunes (LIMOS), David R.C. Hill (ISIMA, LIMOS)
- Abstract summary: Reproducibility is a fundamental principle in scientific research.
Highperformance computing presents unique challenges.
This paper provides a comprehensive review of these concerns and potential solutions.
- Score: 0.0
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Reproducibility is widely acknowledged as a fundamental principle in
scientific research. Currently, the scientific community grapples with numerous
challenges associated with reproducibility, often referred to as the
''reproducibility crisis.'' This crisis permeated numerous scientific
disciplines. In this study, we examined the factors in scientific practices
that might contribute to this lack of reproducibility. Significant focus is
placed on the prevalent integration of computation in research, which can
sometimes function as a black box in published papers. Our study primarily
focuses on highperformance computing (HPC), which presents unique
reproducibility challenges. This paper provides a comprehensive review of these
concerns and potential solutions. Furthermore, we discuss the critical role of
reproducible research in advancing science and identifying persisting issues
within the field of HPC.
Related papers
- Confronting the Reproducibility Crisis: A Case Study in Validating Certified Robustness [0.0]
This paper presents a case study of attempting to validate the results on certified adversarial robustness in "SoK: Certified Robustness for Deep Neural Networks" using the VeriGauge toolkit.
Despite following the documented methodology, numerous software and hardware compatibility issues were encountered, including outdated or unavailable dependencies, version conflicts, and driver incompatibilities.
The paper discusses the broader implications of this crisis, proposing potential solutions such as containerization, software preservation, and comprehensive documentation practices.
arXiv Detail & Related papers (2024-05-29T04:37:19Z) - Targeted Reduction of Causal Models [55.11778726095353]
Causal Representation Learning offers a promising avenue to uncover interpretable causal patterns in simulations.
We introduce Targeted Causal Reduction (TCR), a method for condensing complex intervenable models into a concise set of causal factors.
Its ability to generate interpretable high-level explanations from complex models is demonstrated on toy and mechanical systems.
arXiv Detail & Related papers (2023-11-30T15:46:22Z) - Causal machine learning for single-cell genomics [94.28105176231739]
We discuss the application of machine learning techniques to single-cell genomics and their challenges.
We first present the model that underlies most of current causal approaches to single-cell biology.
We then identify open problems in the application of causal approaches to single-cell data.
arXiv Detail & Related papers (2023-10-23T13:35:24Z) - SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Models [70.5763210869525]
We introduce an expansive benchmark suite SciBench for Large Language Model (LLM)
SciBench contains a dataset featuring a range of collegiate-level scientific problems from mathematics, chemistry, and physics domains.
The results reveal that the current LLMs fall short of delivering satisfactory performance, with the best overall score of merely 43.22%.
arXiv Detail & Related papers (2023-07-20T07:01:57Z) - Computational Reproducibility in Computational Social Science [0.8930269507906258]
We argue that computational-x disciplines such as computational social science are also susceptible for the symptoms of the crises.
We provide solutions for Computational Social Science that hinder researchers from obtaining the highest level of data.
arXiv Detail & Related papers (2023-07-04T21:04:18Z) - A Diachronic Analysis of Paradigm Shifts in NLP Research: When, How, and
Why? [84.46288849132634]
We propose a systematic framework for analyzing the evolution of research topics in a scientific field using causal discovery and inference techniques.
We define three variables to encompass diverse facets of the evolution of research topics within NLP.
We utilize a causal discovery algorithm to unveil the causal connections among these variables using observational data.
arXiv Detail & Related papers (2023-05-22T11:08:00Z) - Reproducibility of Machine Learning: Terminology, Recommendations and
Open Issues [5.30596984761294]
A crisis has been recently acknowledged by scientists and this seems to affect even more Artificial Intelligence and Machine Learning.
We critically review the current literature on the topic and highlight the open issues.
We identify key elements often overlooked in modern Machine Learning and provide novel recommendations for them.
arXiv Detail & Related papers (2023-02-24T15:33:20Z) - Reproducibility in machine learning for medical imaging [3.1390096961027076]
This chapter intends at being an introduction to for researchers in the field of machine learning for medical imaging.
For each of them, we aim at defining it, at describing the requirements to achieve it and at discussing its utility.
The chapter ends with a discussion on the benefits of didactic and with a plea for a non-dogmatic approach to this concept and its implementation in research practice.
arXiv Detail & Related papers (2022-09-12T09:00:04Z) - A Guide to Reproducible Research in Signal Processing and Machine
Learning [9.69596041242667]
In 2016 a survey conducted by the journal Nature found that 50% of researchers were unable to reproduce their own experiments.
We aim to present signal processing researchers with a set of practical tools and strategies that can help mitigate many of the obstacles to producing reproducible computational experiments.
arXiv Detail & Related papers (2021-08-27T16:42:32Z) - Towards Continual Reinforcement Learning: A Review and Perspectives [69.48324517535549]
We aim to provide a literature review of different formulations and approaches to continual reinforcement learning (RL)
While still in its early days, the study of continual RL has the promise to develop better incremental reinforcement learners.
These include applications such as those in the fields of healthcare, education, logistics, and robotics.
arXiv Detail & Related papers (2020-12-25T02:35:27Z) - Heterogeneous Representation Learning: A Review [66.12816399765296]
Heterogeneous Representation Learning (HRL) brings some unique challenges.
We present a unified learning framework which is able to model most existing learning settings with the heterogeneous inputs.
We highlight the challenges that are less-touched in HRL and present future research directions.
arXiv Detail & Related papers (2020-04-28T05:12:31Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.