Research Reproducibility as a Survival Analysis

• URL: http://arxiv.org/abs/2012.09932v1
• Date: Thu, 17 Dec 2020 20:56:53 GMT
• Title: Research Reproducibility as a Survival Analysis
• Authors: Edward Raff
• Abstract summary: We consider modeling a paper as a survival analysis problem. We show how a survival analysis allows us to draw new insights that better explain prior longitudinal data.
• Score: 22.66983713481359
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: There has been increasing concern within the machine learning community that we are in a reproducibility crisis. As many have begun to work on this problem, all work we are aware of treat the issue of reproducibility as an intrinsic binary property: a paper is or is not reproducible. Instead, we consider modeling the reproducibility of a paper as a survival analysis problem. We argue that this perspective represents a more accurate model of the underlying meta-science question of reproducible research, and we show how a survival analysis allows us to draw new insights that better explain prior longitudinal data. The data and code can be found at https://github.com/EdwardRaff/Research-Reproducibility-Survival-Analysis

Related papers

• Can citations tell us about a paper's reproducibility? A case study of machine learning papers [3.5120846057971065]
  Resource constraints and inadequate documentation can make running replications particularly challenging. We introduce a sentiment analysis framework applied to citation contexts from papers involved in Machine Learning Reproducibility Challenges.
  arXiv  Detail & Related papers  (2024-05-07T03:29:11Z)
• Groundedness in Retrieval-augmented Long-form Generation: An Empirical Study [61.74571814707054]
  We evaluate whether every generated sentence is grounded in retrieved documents or the model's pre-training data. Across 3 datasets and 4 model families, our findings reveal that a significant fraction of generated sentences are consistently ungrounded. Our results show that while larger models tend to ground their outputs more effectively, a significant portion of correct answers remains compromised by hallucinations.
  arXiv  Detail & Related papers  (2024-04-10T14:50:10Z)
• Interpretable Survival Analysis for Heart Failure Risk Prediction [50.64739292687567]
  We propose a novel survival analysis pipeline that is both interpretable and competitive with state-of-the-art survival models. Our pipeline achieves state-of-the-art performance and provides interesting and novel insights about risk factors for heart failure.
  arXiv  Detail & Related papers  (2023-10-24T02:56:05Z)
• Repetition In Repetition Out: Towards Understanding Neural Text Degeneration from the Data Perspective [91.14291142262262]
  This work presents a straightforward and fundamental explanation from the data perspective. Our preliminary investigation reveals a strong correlation between the degeneration issue and the presence of repetitions in training data. Our experiments reveal that penalizing the repetitions in training data remains critical even when considering larger model sizes and instruction tuning.
  arXiv  Detail & Related papers  (2023-10-16T09:35:42Z)
• Reproducibility Requires Consolidated Artifacts [14.481126181883814]
  Machine learning is facing a'reproducibility crisis' where a significant number of works report failures when attempting to reproduce previously published results. We evaluate the sources of failures using a meta-analysis of 142 replication studies from ReScience C and 204 code repositories.
  arXiv  Detail & Related papers  (2023-05-21T21:21:46Z)
• Missing Information, Unresponsive Authors, Experimental Flaws: The Impossibility of Assessing the Reproducibility of Previous Human Evaluations in NLP [84.08476873280644]
  Just 13% of papers had (i) sufficiently low barriers to reproduction, and (ii) enough obtainable information, to be considered for reproduction. As a result, we had to change our coordinated study design from a reproduce approach to a standardise-then-reproduce-twice approach.
  arXiv  Detail & Related papers  (2023-05-02T17:46:12Z)
• SurvivalGAN: Generating Time-to-Event Data for Survival Analysis [121.84429525403694]
  Imbalances in censoring and time horizons cause generative models to experience three new failure modes specific to survival analysis. We propose SurvivalGAN, a generative model that handles survival data by addressing the imbalance in the censoring and event horizons. We evaluate this method via extensive experiments on medical datasets.
  arXiv  Detail & Related papers  (2023-02-24T17:03:51Z)
• Reproducibility of Machine Learning: Terminology, Recommendations and Open Issues [5.30596984761294]
  A crisis has been recently acknowledged by scientists and this seems to affect even more Artificial Intelligence and Machine Learning. We critically review the current literature on the topic and highlight the open issues. We identify key elements often overlooked in modern Machine Learning and provide novel recommendations for them.
  arXiv  Detail & Related papers  (2023-02-24T15:33:20Z)
• Reproducibility in machine learning for medical imaging [3.1390096961027076]
  This chapter intends at being an introduction to for researchers in the field of machine learning for medical imaging. For each of them, we aim at defining it, at describing the requirements to achieve it and at discussing its utility. The chapter ends with a discussion on the benefits of didactic and with a plea for a non-dogmatic approach to this concept and its implementation in research practice.
  arXiv  Detail & Related papers  (2022-09-12T09:00:04Z)
• ACRE: Abstract Causal REasoning Beyond Covariation [90.99059920286484]
  We introduce the Abstract Causal REasoning dataset for systematic evaluation of current vision systems in causal induction. Motivated by the stream of research on causal discovery in Blicket experiments, we query a visual reasoning system with the following four types of questions in either an independent scenario or an interventional scenario. We notice that pure neural models tend towards an associative strategy under their chance-level performance, whereas neuro-symbolic combinations struggle in backward-blocking reasoning.
  arXiv  Detail & Related papers  (2021-03-26T02:42:38Z)
• The Fundamental Principles of Reproducibility [2.4671396651514983]
  I take a fundamental view on rooted in the scientific method. The scientific method is analysed and characterised in order to develop the terminology required to define.
  arXiv  Detail & Related papers  (2020-11-19T20:37:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.