AI Research moves towards open and reproducible science

• URL: http://arxiv.org/abs/2606.16974v2
• Date: Mon, 22 Jun 2026 03:57:39 GMT
• Title: AI Research moves towards open and reproducible science
• Abstract summary: The crisis has directed the AI research community toward improving documentation practices.<n>In response, the most impactful venues in the field have introduced checklists.<n>We assess whether documentation practices have changed over time by assessing all published papers at five leading AI conferences over the past decade.
• Score: 3.590867023672793
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The reproducibility crisis has directed the AI research community toward improving documentation practices. Several studies have identified methodological issues, and in response, the most impactful venues in the field have introduced reproducibility checklists. We seek to understand whether documentation practices have changed over time by assessing all published papers at five leading AI conferences over the past decade. Seven reproducibility variables were identified, quality-assured and used to analyse 56 800 publications. Our analysis reveals that in the period 2014 to 2024, documentation practices have improved; papers sharing both code and data increased nearly sixfold, from 11% to 64% Building on empirical reproducibility rates from a prior study, we estimate - inferred from documentation practices, not direct testing - that reproducibility increased from 28% in 2014 to 64% in 2024. Improvements in documentation practices predate the introduction of reproducibility checklists, suggesting these changes reflect a broader movement toward open science rather than a direct response to formal requirements.

Related papers

• PaperGym: Rubric-Centered Evolution for Research-Plan Generation [69.5921878728483]
  We introduce PaperGym, a unified framework that turns each research paper into a complete training environment.<n>PaperGym exploits the structure of a paper: the question is synthesized from the research goal and background, while the criteria are derived from the method and experiments.<n>With the recipe held fixed, models trained on PaperGym-20k win 58.1% of three-way comparisons, against 28.2% forHub Science.
  arXiv  Detail & Related papers  (2026-08-31T17:31:18Z)
• From Interpretability to Control: Insights from Six Years of the TrustNLP Workshop [63.00693129203923]
  The Workshop on Trustworthy Natural Language Processing has grown from 8 proceedings papers to 41 over six editions.<n>We synthesize insights from all 144 proceedings papers, classifying them along six trust dimensions grounded in established frameworks.<n>Analysis from the classification study reveals that truthfulness is the fastest-growing dimension (absent in 2021-2022, comprising 37% of papers by 2025-2026)
  arXiv  Detail & Related papers  (2026-08-11T17:30:16Z)
• ResearchClawBench: A Benchmark for End-to-End Autonomous Scientific Research [142.29356526274387]
  We present ResearchClawBench, a benchmark for evaluating autonomous scientific research.<n>Each task is grounded in a real published paper, provides related literature and raw data, and hides the target paper during evaluation.
  arXiv  Detail & Related papers  (2026-05-28T16:27:40Z)
• AI-Assisted Curation of Conference Scholarship: Compiling, Structuring, and Analyzing Two Decades of Presentations at the Society for Social Work and Research [0.0]
  This study developed a comprehensive database of presentation abstracts from the Society for Social Work and Research (SSWR) Annual Conference.<n> Abstract metadata was compiled from the SSWR Confex conference management system for presentations from 2005 to 2026 using web scraping.<n>The database contains 23,793 presentations with 69,924 author records representing 20,779 unique researchers from 4,049 institutions across 93 countries.
  arXiv  Detail & Related papers  (2026-03-06T19:19:29Z)
• SciCoQA: Quality Assurance for Scientific Paper--Code Alignment [53.70401063640645]
  We present SciCoQA, a dataset for detecting discrepancies between scientific publications and theirs.<n>Our dataset consists of 611 paper-code discrepancies (81 real, 530 synthetic), spanning diverse computational science disciplines.<n>The best performing model in our evaluation, GPT-5, can only detect 45.7% of real-world paper-code discrepancies.
  arXiv  Detail & Related papers  (2026-01-19T10:04:33Z)
• The State of Open Science in Software Engineering Research: A Case Study of ICSE Artifacts [2.5705703401045557]
  There is a marked lack of studies that comprehensively examine the executability and rigor of replication packages in software engineering (SE) research.<n>We evaluate 100 replication packages published as part of ICSE proceedings over the past decade.<n>Our findings reveal that only 40% of the 100 artifacts evaluated were executable, of which 32.5% (13 out of 40) ran without any modification.
  arXiv  Detail & Related papers  (2026-01-05T12:47:43Z)
• Large Language Models for Software Engineering: A Reproducibility Crisis [4.730658148470817]
  This paper presents the first large-scale, empirical study of practices in large language model (LLM)-based software engineering research.<n>We systematically mined and analyzed 640 papers published between 2017 and 2025 across premier software engineering, machine learning, and natural language processing venues.<n>Our analysis reveals persistent gaps in artifact availability, environment specification, versioning rigor, and documentation clarity.
  arXiv  Detail & Related papers  (2025-11-29T22:16:47Z)
• Exploring the change in scientific readability following the release of ChatGPT [2.7195102129095003]
  This dataset consists of all abstracts posted on arXiv between 2010 and June 7th, 2024.<n>The results show a steady annual decrease in readability, suggesting that abstracts are likely becoming increasingly complex.<n>Following the release of ChatGPT, a significant change in readability is observed for 2023 and the analyzed months of 2024.
  arXiv  Detail & Related papers  (2025-06-26T23:57:12Z)
• Reproducibility and Artifact Consistency of the SIGIR 2022 Recommender Systems Papers Based on Message Passing [10.474191156751928]
  We analyze 10 graph-based Recommender Systems papers presented at SIGIR 2022 and 2023.<n>Our analysis reveals several critical points that require attention.<n>Due to these issues, we are unable to confirm the claims made in most of the papers we examined and attempted to reproduce.
  arXiv  Detail & Related papers  (2025-03-10T20:09:04Z)
• CycleResearcher: Improving Automated Research via Automated Review [37.03497673861402]
  This paper explores the possibility of using open-source post-trained large language models (LLMs) as autonomous agents capable of performing the full cycle of automated research and review.<n>To train these models, we develop two new datasets, reflecting real-world machine learning research and peer review dynamics.<n>Our results demonstrate that CycleReviewer achieves promising performance with a 26.89% reduction in mean absolute error (MAE) compared to individual human reviewers in predicting paper scores.
  arXiv  Detail & Related papers  (2024-10-28T08:10:21Z)
• Advancing Question Answering on Handwritten Documents: A State-of-the-Art Recognition-Based Model for HW-SQuAD [30.559280110711143]
  This paper proposes a novel recognition-based approach that improves upon the previous state-of-the-art on the HW-SQuAD and BenthamQA datasets. Our model incorporates transformer-based document retrieval and ensemble methods at the model level, achieving an Exact Match score of 82.02% and 69% in HW-SQuAD and BenthamQA datasets.
  arXiv  Detail & Related papers  (2024-06-25T10:18:50Z)
• SciRIFF: A Resource to Enhance Language Model Instruction-Following over Scientific Literature [97.31347312130119]
  SciRIFF (Scientific Resource for Instruction-Following and Finetuning) is a dataset of 137K instruction-following instances for training and evaluation, covering 54 tasks.<n>These tasks span five core scientific literature understanding capabilities: information extraction, summarization, question answering, claim verification, and classification.<n> SciRIFF is unique in being entirely expert-written, high-quality instruction-following dataset for extracting and synthesizing information from research literature across diverse scientific fields.
  arXiv  Detail & Related papers  (2024-06-10T21:22:08Z)
• Monitoring AI-Modified Content at Scale: A Case Study on the Impact of ChatGPT on AI Conference Peer Reviews [51.453135368388686]
  We present an approach for estimating the fraction of text in a large corpus which is likely to be substantially modified or produced by a large language model (LLM) Our maximum likelihood model leverages expert-written and AI-generated reference texts to accurately and efficiently examine real-world LLM-use at the corpus level.
  arXiv  Detail & Related papers  (2024-03-11T21:51:39Z)
• Reproducibility in NLP: What Have We Learned from the Checklist? [48.86370747121637]
  The *CL conferences created the NLP Reproducibility Checklist in 2020 to be completed by authors at submission to remind them of key information to include. We provide the first analysis of the Checklist by examining 10,405 anonymous responses to it. We find that the 44% of submissions that gather new data are 5% less likely to be accepted than those that did not.
  arXiv  Detail & Related papers  (2023-06-16T00:39:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.