The Unreasonable Effectiveness of Open Science in AI: A Replication Study

• URL: http://arxiv.org/abs/2412.17859v1
• Date: Fri, 20 Dec 2024 12:33:27 GMT
• Title: The Unreasonable Effectiveness of Open Science in AI: A Replication Study
• Abstract summary: A crisis has been reported in science, but the extent to which it affects AI research is not yet fully understood.<n>We performed a systematic replication study including 30 highly cited AI studies relying on original materials when available.<n>Six articles were successfully reproduced, while five were partially reproduced.
• Score: 1.991176263126831
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: A reproducibility crisis has been reported in science, but the extent to which it affects AI research is not yet fully understood. Therefore, we performed a systematic replication study including 30 highly cited AI studies relying on original materials when available. In the end, eight articles were rejected because they required access to data or hardware that was practically impossible to acquire as part of the project. Six articles were successfully reproduced, while five were partially reproduced. In total, 50% of the articles included was reproduced to some extent. The availability of code and data correlate strongly with reproducibility, as 86% of articles that shared code and data were fully or partly reproduced, while this was true for 33% of articles that shared only data. The quality of the data documentation correlates with successful replication. Poorly documented or miss-specified data will probably result in unsuccessful replication. Surprisingly, the quality of the code documentation does not correlate with successful replication. Whether the code is poorly documented, partially missing, or not versioned is not important for successful replication, as long as the code is shared. This study emphasizes the effectiveness of open science and the importance of properly documenting data work.

Related papers

• PaperDoctor: Evidence-Grounded and Actionable Feedback for Scientific Papers in Progress [130.58520199343707]
  PaperDoctor is an agent framework for pre-submission feedback with three key innovations.<n>First, a holistic hierarchical framework evaluates writing, layout, references, code, theory, prior work, and experiments.<n>Second, each finding contains an observation, a pointer to specific evidence such as a sentence, equation, or code line, and a revision suggestion.<n>Third, PaperDoctor selectively rebuilds and reruns experiments based on claim importance and compute budget.
  arXiv  Detail & Related papers  (2026-09-15T11:08:11Z)
• ReproRepo: Scaling Reproducibility Audits with GitHub Repository Issues [54.98461672730087]
  We introduce ReproRepo, a scalable framework for evaluation that leverages human-raised GitHub issues as naturally occurring supervision on realistic reproduction blockers.<n>We instantiate ReproRepo on 1,149 recent machine learning papers from major conferences and evaluate four frontier model-agent configurations.<n>Our results show that LLM agents, even without executing code, can identify many real-world problems from paper-repository pairs.
  arXiv  Detail & Related papers  (2026-06-16T17:58:05Z)
• Beyond Memory: A Templated Substrate for Heterogeneous Collaborative Knowledge Work with LLM Agents [0.0]
  We present llm-wiki-memory-template, a reusable, agent-aware instantiation.<n>It is a substrate for heterogeneous collaborative knowledge work along three axes.<n>The wiki is append-only by convention, which preserves what did not work alongside what did.
  arXiv  Detail & Related papers  (2026-05-29T15:34:14Z)
• FactReview: Evidence-Grounded Reviews with Literature Positioning and Execution-Based Claim Verification [57.196748998757954]
  We present FactReview, an evidence-grounded reviewing system that combines claim extraction, literature positioning, and execution-based claim verification.<n>Given a submission, FactReview identifies major claims and reported results, retrieves nearby work to clarify the paper's technical position, and, when code is available, executes the released repository.<n>It then produces a concise review and an evidence report that assigns each major claim one of five labels: Supported, Supported by the paper, Partially supported, In conflict, or Inconclusive.
  arXiv  Detail & Related papers  (2026-04-05T11:45:22Z)
• What Papers Don't Tell You: Recovering Tacit Knowledge for Automated Paper Reproduction [57.86097956633207]
  method is a graph-based agent framework for generating executable code from academic papers.<n>On an extended ReproduceBench spanning 3 domains, 10 tasks, and 40 recent papers, method achieves an average performance gap of 10.04% against official implementations.
  arXiv  Detail & Related papers  (2026-03-02T12:33:31Z)
• SciCoQA: Quality Assurance for Scientific Paper--Code Alignment [53.70401063640645]
  We present SciCoQA, a dataset for detecting discrepancies between scientific publications and theirs.<n>Our dataset consists of 611 paper-code discrepancies (81 real, 530 synthetic), spanning diverse computational science disciplines.<n>The best performing model in our evaluation, GPT-5, can only detect 45.7% of real-world paper-code discrepancies.
  arXiv  Detail & Related papers  (2026-01-19T10:04:33Z)
• The State of Open Science in Software Engineering Research: A Case Study of ICSE Artifacts [2.5705703401045557]
  There is a marked lack of studies that comprehensively examine the executability and rigor of replication packages in software engineering (SE) research.<n>We evaluate 100 replication packages published as part of ICSE proceedings over the past decade.<n>Our findings reveal that only 40% of the 100 artifacts evaluated were executable, of which 32.5% (13 out of 40) ran without any modification.
  arXiv  Detail & Related papers  (2026-01-05T12:47:43Z)
• VeriCite: Towards Reliable Citations in Retrieval-Augmented Generation via Rigorous Verification [107.75781898355562]
  We introduce a novel framework, called VeriCite, designed to rigorously validate supporting evidence and enhance answer attribution.<n>We conduct experiments across five open-source LLMs and four datasets, demonstrating that VeriCite can significantly improve citation quality while maintaining the correctness of the answers.
  arXiv  Detail & Related papers  (2025-10-13T13:38:54Z)
• Reproducibility of Machine Learning-Based Fault Detection and Diagnosis for HVAC Systems in Buildings: An Empirical Study [7.852209218432359]
  This paper analyzes the transparency and standards of Machine Learning applications in building energy systems.<n>The results indicate that nearly all articles are not reproducible due to insufficient disclosure.<n>These findings highlight the need for targeted interventions, including guidelines, training for researchers, and policies by journals and conferences.
  arXiv  Detail & Related papers  (2025-07-23T07:35:58Z)
• Computational Reproducibility of R Code Supplements on OSF [0.0]
  Many published code supplements lack the necessary documentation to recreate their computational environments.<n>In this work, we assess the computational of 296 R projects using the StatCodeSearch dataset.<n>We developed an automated pipeline that reconstructs computational environments directly from project source code.
  arXiv  Detail & Related papers  (2025-05-27T13:06:38Z)
• AutoReproduce: Automatic AI Experiment Reproduction with Paper Lineage [62.049868205196425]
  AutoReproduce is a framework capable of automatically reproducing experiments described in research papers in an end-to-end manner.<n>Results show that AutoReproduce achieves an average performance gap of $22.1%$ on $89.74%$ of the executable experiment runs.
  arXiv  Detail & Related papers  (2025-05-27T03:15:21Z)
• Collapse of Dense Retrievers: Short, Early, and Literal Biases Outranking Factual Evidence [56.09494651178128]
  Retrieval models are commonly used in Information Retrieval (IR) applications, such as Retrieval-Augmented Generation (RAG) We show that retrievers often rely on superficial patterns like over-prioritizing document beginnings, shorter documents, repeated entities, and literal matches. We show that these biases have direct consequences for downstream applications like RAG, where retrieval-preferred documents can mislead LLMs.
  arXiv  Detail & Related papers  (2025-03-06T23:23:13Z)
• CopyBench: Measuring Literal and Non-Literal Reproduction of Copyright-Protected Text in Language Model Generation [132.00910067533982]
  We introduce CopyBench, a benchmark designed to measure both literal and non-literal copying in LM generations. We find that, although literal copying is relatively rare, two types of non-literal copying -- event copying and character copying -- occur even in models as small as 7B parameters.
  arXiv  Detail & Related papers  (2024-07-09T17:58:18Z)
• Examining Ownership Models in Software Teams: A Systematic Literature Review and a Replication Study [2.0891120283967264]
  We identify 79 relevant papers published between 2005 and 2022. We develop a taxonomy of ownership artifacts based on type, owners, and degree of ownership.
  arXiv  Detail & Related papers  (2024-05-24T16:03:22Z)
• Automatically Finding and Categorizing Replication Studies [0.0]
  In many fields of experimental science, papers that failed to replicate continue to be cited as a result of the poor discoverability of replication studies. As a first step to creating a system that automatically finds replication studies for a given paper, 334 replication studies and 344 replicated studies were collected.
  arXiv  Detail & Related papers  (2023-11-25T15:27:10Z)
• Replication: Contrastive Learning and Data Augmentation in Traffic Classification Using a Flowpic Input Representation [47.95762911696397]
  We reproduce [16] on the same datasets and replicate its most salient aspect (the importance of data augmentation) on three additional public datasets. While we confirm most of the original results, we also found a 20% accuracy drop on some of the investigated scenarios due to a data shift in the original dataset.
  arXiv  Detail & Related papers  (2023-09-18T12:55:09Z)
• In-class Data Analysis Replications: Teaching Students while Testing Science [16.951059542542843]
  In the present study, we incorporated data analysis replications in the project component of the Applied Data Analysis course taught at EPFL. We find discrepancies between what students expect of data analysis replications and what they experience. We identify tangible benefits of the in-class data analysis replications for scientific communities.
  arXiv  Detail & Related papers  (2023-08-31T06:53:22Z)
• A Study on Reproducibility and Replicability of Table Structure Recognition Methods [3.8366337377024298]
  We examine both and replicability of a corpus of 16 papers on table structure recognition (TSR) We reproduce results consistent with the original in only four of the 16 papers studied. No paper is identified as replicable using the new dataset.
  arXiv  Detail & Related papers  (2023-04-20T16:30:58Z)
• Black-box Dataset Ownership Verification via Backdoor Watermarking [67.69308278379957]
  We formulate the protection of released datasets as verifying whether they are adopted for training a (suspicious) third-party model. We propose to embed external patterns via backdoor watermarking for the ownership verification to protect them. Specifically, we exploit poison-only backdoor attacks ($e.g.$, BadNets) for dataset watermarking and design a hypothesis-test-guided method for dataset verification.
  arXiv  Detail & Related papers  (2022-08-04T05:32:20Z)
• Long-term Reproducibility for Neural Architecture Search [4.301276597844756]
  There is no academic 'kudos' for bug fixes / maintenance. Code is often unavailable or, if available, contains bugs, is incomplete, or relies on out-of-date / unavailable libraries.
  arXiv  Detail & Related papers  (2022-07-11T12:40:26Z)
• Enhancing Scientific Papers Summarization with Citation Graph [78.65955304229863]
  We redefine the task of scientific papers summarization by utilizing their citation graph. We construct a novel scientific papers summarization dataset Semantic Scholar Network (SSN) which contains 141K research papers in different domains. Our model can achieve competitive performance when compared with the pretrained models.
  arXiv  Detail & Related papers  (2021-04-07T11:13:35Z)
• Can We Automate Scientific Reviewing? [89.50052670307434]
  We discuss the possibility of using state-of-the-art natural language processing (NLP) models to generate first-pass peer reviews for scientific papers. We collect a dataset of papers in the machine learning domain, annotate them with different aspects of content covered in each review, and train targeted summarization models that take in papers to generate reviews. Comprehensive experimental results show that system-generated reviews tend to touch upon more aspects of the paper than human-written reviews.
  arXiv  Detail & Related papers  (2021-01-30T07:16:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.