Containing the Reproducibility Gap: Automated Repository-Level Containerization for Scholarly Jupyter Notebooks

• URL: http://arxiv.org/abs/2604.01072v1
• Date: Wed, 01 Apr 2026 16:07:54 GMT
• Title: Containing the Reproducibility Gap: Automated Repository-Level Containerization for Scholarly Jupyter Notebooks
• Abstract summary: Environment drift, undocumented dependencies and implicit execution assumptions prevent independent re-execution of published research.<n>We present an automated, web-oriented engineering pipeline that reconstructs and evaluates repository-level execution environments for scholarly notebooks.<n>System performs dependency inference, automated container generation, and isolated execution to approximate the notebook's original computational context.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Computational reproducibility is fundamental to trustworthy science, yet remains difficult to achieve in practice across various research workflows, including Jupyter notebooks published alongside scholarly articles. Environment drift, undocumented dependencies and implicit execution assumptions frequently prevent independent re-execution of published research. Despite existing reproducibility guidelines, scalable and systematic infrastructure for automated assessment remains limited. We present an automated, web-oriented reproducibility engineering pipeline that reconstructs and evaluates repository-level execution environments for scholarly notebooks. The system performs dependency inference, automated container generation, and isolated execution to approximate the notebook's original computational context. We evaluate the approach on 443 notebooks from 116 GitHub repositories referenced by publications in PubMed Central. Execution outcomes are classified into four categories: resolved environment failures, persistent logic or data errors, reproducibility drift, and container-induced regressions. Our results show that containerization resolves 66.7% of prior dependency-related failures and substantially improves execution robustness. However, a significant reproducibility gap remains: 53.7% of notebooks exhibit low output fidelity, largely due to persistent runtime failures and stochastic non-determinism. These findings indicate that standardized containerization is essential for computational stability but insufficient for full bit-wise reproducibility. The framework offers a scalable solution for researchers, editors, and archivists seeking systematic, automated assessment of computational artifacts.

Related papers

• From Interaction Traces to Persistent Skills: Online Evolution for Computer-Use Agents [48.50635499112162]
  We present an online skill-evolution framework that converts interaction trajectories and evaluator feedback into a persistent, versioned library of reusable procedures.<n>We compare the full evolving-library system with a configuration-matched empty-library control across four OSWorld application domains.<n>In GIMP, provenance-aware analysis reveals retrieval across task-of-origin boundaries and revision, where repeated accepted edits fail to recover the originating task.
  arXiv  Detail & Related papers  (2026-09-04T08:29:45Z)
• Works on My QPU: Reproducibility in Quantum Computing Research [1.0482211230201115]
  We present a combined manual and automated large-scale analysis of the landscape in quantum computing research.<n>We manually evaluate a curated sample of 127 papers using a five-question framework that covers code availability, environment specification, documentation, hardware description, and executability.<n>Our manual analysis reveals that only 24.4% of the sampled papers provide code artefacts, and among those, 64.5% fail to execute successfully in a clean environment.
  arXiv  Detail & Related papers  (2026-07-09T10:53:02Z)
• One Reflection Is Not Enough: Self-Correcting Autonomous Research via Multi-Hypothesis Failure Attribution [85.65263343588026]
  We propose SAGE, a Self-correcting, Autonomous, Grounded Experimenter.<n>Its core mechanism, Multi-Hypothesis Failure Attribution (MHFA), treats recovery as a structured causal diagnosis.<n>On a 12-topic, 5-domain benchmark, SAGE increases metrics-bearing outputs from 42% to 92% over a baseline reflection.
  arXiv  Detail & Related papers  (2026-06-30T10:54:16Z)
• AI for Auto-Research: Roadmap & User Guide [107.0834449839233]
  We present an end-to-end analysis of AI across the complete research lifecycle.<n>We identify a sharp, stage-dependent boundary between reliable assistance and unreliable autonomy.<n>We show that greater automation can obscure rather than eliminate failure modes.
  arXiv  Detail & Related papers  (2026-05-18T17:08:26Z)
• ReproScore: Separating Readiness from Outcome in Research Software Reproducibility Assessment [0.9623578875486184]
  Digital libraries curate millions of research software artefacts yet lack scalable infrastructure for assessing whether those artefacts remain executable.<n>We present ReproScore, a two-tier framework that explicitly separates readiness (RRS) from outcome (ROS)<n>RRS comprises 26 sub-metrics across five categories; ROS provides execution-based probes when sandbox infrastructure is available; a community externalises weighting priorities as versioned YAML profiles.
  arXiv  Detail & Related papers  (2026-05-13T09:54:06Z)
• The Last Human-Written Paper: Agent-Native Research Artifacts [106.47848184955576]
  We introduce the Agent-Native Research Artifact (ARA), a protocol that replaces the narrative paper with a machine-executable research package.<n>Three mechanisms support the ecosystem: a Live Research Manager that captures decisions and dead ends during ordinary development; an ARA Compiler that translates legacy PDFs and repos into ARAs; and an ARA-native review system that automates objective checks so human reviewers can focus on significance, novelty, and taste.
  arXiv  Detail & Related papers  (2026-04-27T16:23:09Z)
• Dynamic analysis enhances issue resolution [53.50448142467294]
  DAIRA (Dynamic Analysis-enhanced Issue Resolution Agent) is an automated repair framework that embeds dynamic analysis into the agent's reasoning cycle.<n> Driven by a Test Tracing-Driven methodology, DAIRA utilizes lightweight monitors to extract critical runtime data.<n>When powered by Gemini 3 Flash Preview, DAIRA establishes a new state-of-the-art (SOTA) performance, achieving a 79.4% resolution rate on the SWE-bench Verified dataset.
  arXiv  Detail & Related papers  (2026-03-23T14:48:54Z)
• Automated Self-Testing as a Quality Gate: Evidence-Driven Release Management for LLM Applications [51.56484100374058]
  We present an automated self-testing framework that introduces quality gates with evidence-based release decisions.<n>We evaluate the framework through a longitudinal case study of an internally deployed multi-agent conversational AI system.
  arXiv  Detail & Related papers  (2026-03-13T20:44:15Z)
• PaperRepro: Automated Computational Reproducibility Assessment for Social Science Papers [33.12402746591649]
  PaperRepro is a novel two-stage, multi-agent approach for automated assessment.<n>In the execution stage, agents execute the reproduction package and edit the code to capture reproduced results as explicit artifacts.<n>In the evaluation stage, agents evaluate agents using explicit evidence.
  arXiv  Detail & Related papers  (2026-02-10T09:04:59Z)
• NL2Repo-Bench: Towards Long-Horizon Repository Generation Evaluation of Coding Agents [79.29376673236142]
  Existing benchmarks fail to rigorously evaluate the long-horizon capabilities required to build complete software systems.<n>We present NL2Repo Bench, a benchmark explicitly designed to evaluate the long-horizon repository generation ability of coding agents.
  arXiv  Detail & Related papers  (2025-12-14T15:12:13Z)
• RARE: Retrieval-Aware Robustness Evaluation for Retrieval-Augmented Generation Systems [33.389969814185214]
  Retrieval-Augmented Generation (RAG) enhances recency and factuality in answers.<n>Existing evaluations rarely test how well RAG systems cope with real-world noise, conflicting between internal and external retrieved contexts, or fast-changing facts.<n>We introduce Retrieval-Aware Robustness Evaluation (RARE), a unified framework and large-scale benchmark that jointly stress-test query and document perturbations over dynamic, time-sensitive corpora.
  arXiv  Detail & Related papers  (2025-06-01T02:42:36Z)
• Computational Reproducibility of R Code Supplements on OSF [0.0]
  Many published code supplements lack the necessary documentation to recreate their computational environments.<n>In this work, we assess the computational of 296 R projects using the StatCodeSearch dataset.<n>We developed an automated pipeline that reconstructs computational environments directly from project source code.
  arXiv  Detail & Related papers  (2025-05-27T13:06:38Z)
• AutoReproduce: Automatic AI Experiment Reproduction with Paper Lineage [62.049868205196425]
  AutoReproduce is a framework capable of automatically reproducing experiments described in research papers in an end-to-end manner.<n>Results show that AutoReproduce achieves an average performance gap of $22.1%$ on $89.74%$ of the executable experiment runs.
  arXiv  Detail & Related papers  (2025-05-27T03:15:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.