Reproducibility Requires Consolidated Artifacts

• URL: http://arxiv.org/abs/2305.12571v1
• Date: Sun, 21 May 2023 21:21:46 GMT
• Title: Reproducibility Requires Consolidated Artifacts
• Authors: Iordanis Fostiropoulos, Bowman Brown, Laurent Itti
• Abstract summary: Machine learning is facing a'reproducibility crisis' where a significant number of works report failures when attempting to reproduce previously published results. We evaluate the sources of failures using a meta-analysis of 142 replication studies from ReScience C and 204 code repositories.
• Score: 14.481126181883814
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Machine learning is facing a 'reproducibility crisis' where a significant number of works report failures when attempting to reproduce previously published results. We evaluate the sources of reproducibility failures using a meta-analysis of 142 replication studies from ReScience C and 204 code repositories. We find that missing experiment details such as hyperparameters are potential causes of unreproducibility. We experimentally show the bias of different hyperparameter selection strategies and conclude that consolidated artifacts with a unified framework can help support reproducibility.

Related papers

• Continuous Analysis: Evolution of Software Engineering and Reproducibility for Science [0.0]
  This paper introduces the concept of Continuous Analysis to address the challenges in scientific research. By adopting CA, the scientific community can ensure the validity and generalizability of research outcomes.
  arXiv  Detail & Related papers  (2024-11-04T17:11:08Z)
• Can citations tell us about a paper's reproducibility? A case study of machine learning papers [3.5120846057971065]
  Resource constraints and inadequate documentation can make running replications particularly challenging. We introduce a sentiment analysis framework applied to citation contexts from papers involved in Machine Learning Reproducibility Challenges.
  arXiv  Detail & Related papers  (2024-05-07T03:29:11Z)
• Multi-modal Causal Structure Learning and Root Cause Analysis [67.67578590390907]
  We propose Mulan, a unified multi-modal causal structure learning method for root cause localization. We leverage a log-tailored language model to facilitate log representation learning, converting log sequences into time-series data. We also introduce a novel key performance indicator-aware attention mechanism for assessing modality reliability and co-learning a final causal graph.
  arXiv  Detail & Related papers  (2024-02-04T05:50:38Z)
• Identifiable Latent Polynomial Causal Models Through the Lens of Change [82.14087963690561]
  Causal representation learning aims to unveil latent high-level causal representations from observed low-level data. One of its primary tasks is to provide reliable assurance of identifying these latent causal models, known as identifiability.
  arXiv  Detail & Related papers  (2023-10-24T07:46:10Z)
• Advancing Counterfactual Inference through Nonlinear Quantile Regression [77.28323341329461]
  We propose a framework for efficient and effective counterfactual inference implemented with neural networks. The proposed approach enhances the capacity to generalize estimated counterfactual outcomes to unseen data. Empirical results conducted on multiple datasets offer compelling support for our theoretical assertions.
  arXiv  Detail & Related papers  (2023-06-09T08:30:51Z)
• Missing Information, Unresponsive Authors, Experimental Flaws: The Impossibility of Assessing the Reproducibility of Previous Human Evaluations in NLP [84.08476873280644]
  Just 13% of papers had (i) sufficiently low barriers to reproduction, and (ii) enough obtainable information, to be considered for reproduction. As a result, we had to change our coordinated study design from a reproduce approach to a standardise-then-reproduce-twice approach.
  arXiv  Detail & Related papers  (2023-05-02T17:46:12Z)
• Boosting Differentiable Causal Discovery via Adaptive Sample Reweighting [62.23057729112182]
  Differentiable score-based causal discovery methods learn a directed acyclic graph from observational data. We propose a model-agnostic framework to boost causal discovery performance by dynamically learning the adaptive weights for the Reweighted Score function, ReScore.
  arXiv  Detail & Related papers  (2023-03-06T14:49:59Z)
• Returning The Favour: When Regression Benefits From Probabilistic Causal Knowledge [9.106412307976067]
  A directed acyclic graph (DAG) provides valuable prior knowledge that is often discarded in regression tasks in machine learning. We show that the independences arising from the presence of collider structures in DAGs provide meaningful inductive biases, which constrain the regression hypothesis space and improve predictive performance.
  arXiv  Detail & Related papers  (2023-01-26T16:44:15Z)
• The worst of both worlds: A comparative analysis of errors in learning from data in psychology and machine learning [17.336655978572583]
  Recent concerns that machine learning (ML) may be facing a misdiagnosis and replication crisis suggest that some published claims in ML research cannot be taken at face value. A deeper understanding of what concerns in research in supervised ML have in common with the replication crisis in experimental science can put the new concerns in perspective.
  arXiv  Detail & Related papers  (2022-03-12T18:26:24Z)
• Discovering Latent Causal Variables via Mechanism Sparsity: A New Principle for Nonlinear ICA [81.4991350761909]
  Independent component analysis (ICA) refers to an ensemble of methods which formalize this goal and provide estimation procedure for practical application. We show that the latent variables can be recovered up to a permutation if one regularizes the latent mechanisms to be sparse.
  arXiv  Detail & Related papers  (2021-07-21T14:22:14Z)
• Research Reproducibility as a Survival Analysis [22.66983713481359]
  We consider modeling a paper as a survival analysis problem. We show how a survival analysis allows us to draw new insights that better explain prior longitudinal data.
  arXiv  Detail & Related papers  (2020-12-17T20:56:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.