A Study of Scientific Computational Notebook Quality

• URL: http://arxiv.org/abs/2603.22726v1
• Date: Tue, 24 Mar 2026 02:44:57 GMT
• Title: A Study of Scientific Computational Notebook Quality
• Abstract summary: Poorly written code can result in incorrect findings, and slower scientific progress.<n>We evaluate scientific code quality across three dimensions: repositories, readability, and reusability.<n>Results reveal major challenges in scientific code quality.
• Score: 1.4853037403155838
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The quality of scientific code is a critical concern for the research community. Poorly written code can result in irreproducible results, incorrect findings, and slower scientific progress. In this study, we evaluate scientific code quality across three dimensions: reproducibility, readability, and reusability. We curated a corpus of 518 code repositories by analyzing Code Availability statements from all 1239 Nature publications in 2024. To assess code quality, we employed multiple methods, including manual attempts to reproduce Jupyter notebooks, documentation reviews, and analyses of code clones and mutation patterns. Our results reveal major challenges in scientific code quality. Of the 19 notebooks we attempted to execute, only two were reproducible, primarily due to missing data files and dependency issues. Code duplication was also common, with 326 clone classes of at least 10 lines and three instances found among 637 of the 1510 notebooks in our corpus. These duplications frequently involved tasks such as visualization, data processing, and statistical analysis. Moreover, our mutation analysis showed that scientific notebooks often exhibit tangled state changes, complicating comprehension and reasoning. The prevalence of these issues -- unreproducible code, widespread duplication, and tangled state management -- underscores the need for improved tools and abstractions to help science build reproducible, readable and reusable software.

Related papers

• PaperDoctor: Evidence-Grounded and Actionable Feedback for Scientific Papers in Progress [130.58520199343707]
  PaperDoctor is an agent framework for pre-submission feedback with three key innovations.<n>First, a holistic hierarchical framework evaluates writing, layout, references, code, theory, prior work, and experiments.<n>Second, each finding contains an observation, a pointer to specific evidence such as a sentence, equation, or code line, and a revision suggestion.<n>Third, PaperDoctor selectively rebuilds and reruns experiments based on claim importance and compute budget.
  arXiv  Detail & Related papers  (2026-09-15T11:08:11Z)
• Can LLMs Discover Scientific Laws in Real and Parallel Worlds? [58.76147897928744]
  We introduce SCILAWS-BENCH, a benchmark for scientific law discovery built from published research and real scientific data.<n>It comprises 118 problems drawn from 381 scientific papers, covering 291 candidate laws and roughly 8M real data points across six scientific disciplines.<n>We find that predictive fit can diverge from scientific validity, memorization shapes whether models reproduce or move beyond published formulas, and our best-of-N study reveals a selection bottleneck.
  arXiv  Detail & Related papers  (2026-09-01T17:17:59Z)
• SWE-bench Science: Can Coding Agents Resolve Engineering Tasks in Science? [60.11366724565423]
  SWE-bench Science is a repository-level benchmark for scientific software engineering comprising 119 tasks from 98 GitHub repositories across 20 scientific domains.<n>We identify four recurring failure mechanisms: deficits in scientific knowledge or abstraction, misguided exploration or surface-level repair, incomplete repair coverage or system integration, and failures to generalize scientific knowledge beyond observed cases in our analysis.
  arXiv  Detail & Related papers  (2026-08-20T08:53:15Z)
• Comparing the Quality of Code Generated by Vibe Coding Tools [6.254624232861127]
  This study compares the structural quality of code produced by three widely adopted vibe coding tools.<n>We generate three independent projects per tool, totalling nine web applications, and submit them to static analysis with SonarQube.<n>Preliminary results show that the tools exhibit distinct qualitative profiles.
  arXiv  Detail & Related papers  (2026-08-17T09:14:05Z)
• Works on My QPU: Reproducibility in Quantum Computing Research [1.0482211230201115]
  We present a combined manual and automated large-scale analysis of the landscape in quantum computing research.<n>We manually evaluate a curated sample of 127 papers using a five-question framework that covers code availability, environment specification, documentation, hardware description, and executability.<n>Our manual analysis reveals that only 24.4% of the sampled papers provide code artefacts, and among those, 64.5% fail to execute successfully in a clean environment.
  arXiv  Detail & Related papers  (2026-07-09T10:53:02Z)
• SciCoQA: Quality Assurance for Scientific Paper--Code Alignment [53.70401063640645]
  We present SciCoQA, a dataset for detecting discrepancies between scientific publications and theirs.<n>Our dataset consists of 611 paper-code discrepancies (81 real, 530 synthetic), spanning diverse computational science disciplines.<n>The best performing model in our evaluation, GPT-5, can only detect 45.7% of real-world paper-code discrepancies.
  arXiv  Detail & Related papers  (2026-01-19T10:04:33Z)
• A Dataset For Computational Reproducibility [2.147712260420443]
  This article introduces a dataset of computational experiments covering a broad spectrum of scientific fields.<n>It incorporates details about software dependencies, execution steps, and configurations necessary for accurate reproduction.<n>It provides a universal benchmark by establishing a standardized dataset for objectively evaluating and comparing the effectiveness of tools.
  arXiv  Detail & Related papers  (2025-04-11T16:45:10Z)
• CodeScientist: End-to-End Semi-Automated Scientific Discovery with Code-based Experimentation [48.12054700748627]
  We introduce CodeScientist, a novel ASD system that frames ideation and experiment construction as a form of genetic search jointly.<n>We use this paradigm to conduct hundreds of automated experiments on machine-generated ideas broadly in the domain of agents and virtual environments.
  arXiv  Detail & Related papers  (2025-03-20T22:37:17Z)
• Exploring Code Comprehension in Scientific Programming: Preliminary Insights from Research Scientists [6.2329239454115415]
  This study surveys 57 research scientists from various disciplines to explore their programming backgrounds, practices, and challenges they face regarding code readability.<n>Scientists mainly use Python and R, relying on documentation for readability.<n>Our findings show low adoption of code quality tools and a trend towards utilizing large language models to improve code quality.
  arXiv  Detail & Related papers  (2025-01-17T08:47:29Z)
• Understanding Code Understandability Improvements in Code Reviews [79.16476505761582]
  We analyzed 2,401 code review comments from Java open-source projects on GitHub. 83.9% of suggestions for improvement were accepted and integrated, with fewer than 1% later reverted.
  arXiv  Detail & Related papers  (2024-10-29T12:21:23Z)
• SciCode: A Research Coding Benchmark Curated by Scientists [37.900374175754465]
  Since language models (LMs) now outperform average humans on many challenging tasks, it has become increasingly difficult to develop challenging, high-quality, and realistic evaluations. We created a scientist-curated coding benchmark, SciCode, which includes problems in mathematics, physics, chemistry, biology, and materials science. Claude3.5-Sonnet, the best-performing model among those tested, can solve only 4.6% of the problems in the most realistic setting.
  arXiv  Detail & Related papers  (2024-07-18T05:15:24Z)
• What's Wrong with Your Code Generated by Large Language Models? An Extensive Study [92.62952504133926]
  This study evaluated the performance of three leading closed-source LLMs and six popular open-source LLMs on three commonly used benchmarks.<n>We developed a taxonomy of bugs for incorrect codes and analyzed the root cause for common bug types.<n>We propose a novel training-free iterative method that introduces self-critique, enabling LLMs to critique and correct their generated code.
  arXiv  Detail & Related papers  (2024-07-08T17:27:17Z)
• Investigating Reproducibility in Deep Learning-Based Software Fault Prediction [16.25827159504845]
  With the rapid adoption of increasingly complex machine learning models, it becomes more and more difficult for scholars to reproduce the results that are reported in the literature. This is in particular the case when the applied deep learning models and the evaluation methodology are not properly documented and when code and data are not shared. We have conducted a systematic review of the current literature and examined the level of 56 research articles that were published between 2019 and 2022 in top-tier software engineering conferences.
  arXiv  Detail & Related papers  (2024-02-08T13:00:18Z)
• What's New? Summarizing Contributions in Scientific Literature [85.95906677964815]
  We introduce a new task of disentangled paper summarization, which seeks to generate separate summaries for the paper contributions and the context of the work. We extend the S2ORC corpus of academic articles by adding disentangled "contribution" and "context" reference labels. We propose a comprehensive automatic evaluation protocol which reports the relevance, novelty, and disentanglement of generated outputs.
  arXiv  Detail & Related papers  (2020-11-06T02:23:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.