Computational Reproducibility of R Code Supplements on OSF

• URL: http://arxiv.org/abs/2505.21590v1
• Date: Tue, 27 May 2025 13:06:38 GMT
• Title: Computational Reproducibility of R Code Supplements on OSF
• Abstract summary: Many published code supplements lack the necessary documentation to recreate their computational environments.<n>In this work, we assess the computational of 296 R projects using the StatCodeSearch dataset.<n>We developed an automated pipeline that reconstructs computational environments directly from project source code.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Computational reproducibility is fundamental to scientific research, yet many published code supplements lack the necessary documentation to recreate their computational environments. While researchers increasingly share code alongside publications, the actual reproducibility of these materials remains poorly understood. In this work, we assess the computational reproducibility of 296 R projects using the StatCodeSearch dataset. Of these, only 264 were still retrievable, and 98.8% lacked formal dependency descriptions required for successful execution. To address this, we developed an automated pipeline that reconstructs computational environments directly from project source code. Applying this pipeline, we executed the R scripts within custom Docker containers and found that 25.87% completed successfully without error. We conducted a detailed analysis of execution failures, identifying reproducibility barriers such as undeclared dependencies, invalid file paths, and system-level issues. Our findings show that automated dependency inference and containerisation can support scalable verification of computational reproducibility and help identify practical obstacles to code reuse and transparency in scientific research.

Related papers

• Adaptive Critical Token-Aware Retrieval for Repository-Level Code Generation [53.85136813868056]
  ACToR is an adaptive critical token-aware retrieval framework for repository-level code generation.<n>We show that ACToR consistently outperforms state-of-the-art methods on repository-level benchmarks.
  arXiv  Detail & Related papers  (2026-09-01T17:59:39Z)
• Works on My QPU: Reproducibility in Quantum Computing Research [1.0482211230201115]
  We present a combined manual and automated large-scale analysis of the landscape in quantum computing research.<n>We manually evaluate a curated sample of 127 papers using a five-question framework that covers code availability, environment specification, documentation, hardware description, and executability.<n>Our manual analysis reveals that only 24.4% of the sampled papers provide code artefacts, and among those, 64.5% fail to execute successfully in a clean environment.
  arXiv  Detail & Related papers  (2026-07-09T10:53:02Z)
• ReproRepo: Scaling Reproducibility Audits with GitHub Repository Issues [54.98461672730087]
  We introduce ReproRepo, a scalable framework for evaluation that leverages human-raised GitHub issues as naturally occurring supervision on realistic reproduction blockers.<n>We instantiate ReproRepo on 1,149 recent machine learning papers from major conferences and evaluate four frontier model-agent configurations.<n>Our results show that LLM agents, even without executing code, can identify many real-world problems from paper-repository pairs.
  arXiv  Detail & Related papers  (2026-06-16T17:58:05Z)
• AI Coding Agents Can Reproduce Social Science Findings [5.410553210073652]
  Recent evidence suggests that AI coding agents can reproduce published findings when provided with original data and code.<n>We introduce SocSci-Repro-Bench, a benchmark of 221 tasks spanning four disciplines and 13 substantive domains.<n>We find that both can reproduce a large share of social science findings, with Claude Code substantially outperforming Codex.
  arXiv  Detail & Related papers  (2026-06-09T21:00:05Z)
• ReproScore: Separating Readiness from Outcome in Research Software Reproducibility Assessment [0.9623578875486184]
  Digital libraries curate millions of research software artefacts yet lack scalable infrastructure for assessing whether those artefacts remain executable.<n>We present ReproScore, a two-tier framework that explicitly separates readiness (RRS) from outcome (ROS)<n>RRS comprises 26 sub-metrics across five categories; ROS provides execution-based probes when sandbox infrastructure is available; a community externalises weighting priorities as versioned YAML profiles.
  arXiv  Detail & Related papers  (2026-05-13T09:54:06Z)
• Can Coding Agents Reproduce Findings in Computational Materials Science? [49.254975563645786]
  We present AutoMat, a benchmark for evaluating large language models' ability to reproduce scientific claims.<n>By working closely with subject matter experts, we curate a set of claims from real materials science papers to test whether coding agents can recover and execute the end-to-end workflow.<n>Results show that current LLM-based agents obtain low overall success rates on AutoMat, with the best-performing setting achieving a success rate of only 54.1%.
  arXiv  Detail & Related papers  (2026-05-01T17:42:12Z)
• Containing the Reproducibility Gap: Automated Repository-Level Containerization for Scholarly Jupyter Notebooks [0.0]
  Environment drift, undocumented dependencies and implicit execution assumptions prevent independent re-execution of published research.<n>We present an automated, web-oriented engineering pipeline that reconstructs and evaluates repository-level execution environments for scholarly notebooks.<n>System performs dependency inference, automated container generation, and isolated execution to approximate the notebook's original computational context.
  arXiv  Detail & Related papers  (2026-04-01T16:07:54Z)
• Understanding by Reconstruction: Reversing the Software Development Process for LLM Pretraining [66.89012795621349]
  Large Language Models (LLMs) often struggle with the deep, long-horizon reasoning required for complex software engineering.<n>We propose a novel paradigm: understanding via reconstruction.<n>We introduce a framework that synthesizes latent agentic trajectories using a multi-agent simulation.
  arXiv  Detail & Related papers  (2026-03-11T09:23:20Z)
• PaperRepro: Automated Computational Reproducibility Assessment for Social Science Papers [33.12402746591649]
  PaperRepro is a novel two-stage, multi-agent approach for automated assessment.<n>In the execution stage, agents execute the reproduction package and edit the code to capture reproduced results as explicit artifacts.<n>In the evaluation stage, agents evaluate agents using explicit evidence.
  arXiv  Detail & Related papers  (2026-02-10T09:04:59Z)
• Automating Computational Reproducibility in Social Science: Comparing Prompt-Based and Agent-Based Approaches [0.3637090866781549]
  Reproducing computational research is often assumed to be as simple as rerunning the original code with provided data.<n>This study investigates whether large language models and AI agents can automate the diagnosis and repair of such failures.
  arXiv  Detail & Related papers  (2026-02-09T11:59:59Z)
• Assessing Reproducibility in Evolutionary Computation: A Case Study using Human- and LLM-based Assessment [2.0365636651755263]
  We study the practices in papers published in the Combinatorial Optimization and Metaheuristics track of the Evolutionary Computation Conference over a ten-year period.<n>We introduce a structured checklist and apply it through a systematic manual assessment of the selected corpus.<n>In addition, we propose RECAP (REproducibility Checklist Automation Pipeline), an automated system that automatically evaluates signals from paper text and associated code.
  arXiv  Detail & Related papers  (2026-02-05T08:32:29Z)
• ABC-Bench: Benchmarking Agentic Backend Coding in Real-World Development [72.4729759618632]
  We introduce ABC-Bench, a benchmark to evaluate agentic backend coding within a realistic, executable workflow.<n>We curated 224 practical tasks spanning 8 languages and 19 frameworks from open-source repositories.<n>Our evaluation reveals that even state-of-the-art models struggle to deliver reliable performance on these holistic tasks.
  arXiv  Detail & Related papers  (2026-01-16T08:23:52Z)
• NL2Repo-Bench: Towards Long-Horizon Repository Generation Evaluation of Coding Agents [79.29376673236142]
  Existing benchmarks fail to rigorously evaluate the long-horizon capabilities required to build complete software systems.<n>We present NL2Repo Bench, a benchmark explicitly designed to evaluate the long-horizon repository generation ability of coding agents.
  arXiv  Detail & Related papers  (2025-12-14T15:12:13Z)
• DeepCode: Open Agentic Coding [11.7906174865581]
  DeepCode is a fully autonomous framework for document-to-codebase synthesis.<n>It orchestrates four information operations to maximize task-relevant signals under finite context budgets.<n>Extensive evaluations on the PaperBench benchmark demonstrate that DeepCode achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2025-12-08T16:07:13Z)
• Taming Imperfect Process Verifiers: A Sampling Perspective on Backtracking [54.43083499412643]
  Test-time algorithms that combine the generative power of language models with process verifiers offer a promising lever for eliciting new reasoning capabilities.<n>We introduce a new process-guided test-time sampling algorithm, VGB, which uses theoretically grounded backtracking to achieve provably better robustness to verifier errors.
  arXiv  Detail & Related papers  (2025-10-03T16:21:14Z)
• AI Copilots for Reproducibility in Science: A Case Study [2.9057981978152116]
  This paper introduces OpenPub, an AI-powered platform that supports researchers, reviewers, and readers through a suite of modular copilots.<n>In this work, we present the Reproducibility Copilot, which analyzes manuscripts, code, and supplementary materials to generate structured Jupyter Notebooks.<n>Results indicate that OpenPub can substantially reduce reproduction time while achieving high coverage of figures, tables, and results suitable for computational reproduction.
  arXiv  Detail & Related papers  (2025-06-25T04:56:28Z)
• AutoReproduce: Automatic AI Experiment Reproduction with Paper Lineage [64.11446484075157]
  AutoReproduce is a framework capable of automatically reproducing experiments described in research papers in an end-to-end manner.<n>Results show that AutoReproduce achieves an average performance gap of $22.1%$ on $89.74%$ of the executable experiment runs.
  arXiv  Detail & Related papers  (2025-05-27T03:15:21Z)
• SciReplicate-Bench: Benchmarking LLMs in Agent-driven Algorithmic Reproduction from Research Papers [16.80818230868491]
  This study evaluates large language models (LLMs) in generating code from algorithm descriptions from recent NLP papers.<n>To facilitate rigorous evaluation, we introduce SciReplicate-Bench, a benchmark of 100 tasks from 36 NLP papers published in 2024.<n>Building on SciReplicate-Bench, we propose Sci-Reproducer, a multi-agent framework consisting of a Paper Agent that interprets algorithmic concepts from literature and a Code Agent that retrieves dependencies from repositories and implement solutions.
  arXiv  Detail & Related papers  (2025-03-31T22:02:24Z)
• Can Language Models Replace Programmers for Coding? REPOCOD Says 'Not Yet' [9.48622608877252]
  A number of repository-level code generation benchmarks have emerged to evaluate the capabilities of large language models (LLMs)<n>These benchmarks consist of short completions, synthetic examples, or focus on limited scale repositories, failing to represent real-world coding tasks.<n>We create REPOCOD, a Python code-generation benchmark containing complex tasks with realistic dependencies in real-world large projects.
  arXiv  Detail & Related papers  (2024-10-29T01:21:05Z)
• Codev-Bench: How Do LLMs Understand Developer-Centric Code Completion? [60.84912551069379]
  We present the Code-Development Benchmark (Codev-Bench), a fine-grained, real-world, repository-level, and developer-centric evaluation framework. Codev-Agent is an agent-based system that automates repository crawling, constructs execution environments, extracts dynamic calling chains from existing unit tests, and generates new test samples to avoid data leakage.
  arXiv  Detail & Related papers  (2024-10-02T09:11:10Z)
• Tractable Offline Learning of Regular Decision Processes [50.11277112628193]
  This work studies offline Reinforcement Learning (RL) in a class of non-Markovian environments called Regular Decision Processes (RDPs) Ins, the unknown dependency of future observations and rewards from the past interactions can be captured experimentally. Many algorithms first reconstruct this unknown dependency using automata learning techniques.
  arXiv  Detail & Related papers  (2024-09-04T14:26:58Z)
• Hierarchical Context Pruning: Optimizing Real-World Code Completion with Repository-Level Pretrained Code LLMs [24.00351065427465]
  We propose a strategy named Hierarchical Context Pruning (HCP) to construct completion prompts with high informational code content. The HCP models the code repository at the function level, maintaining the topological dependencies between code files while removing a large amount of irrelevant code content.
  arXiv  Detail & Related papers  (2024-06-26T12:26:16Z)
• On the Impacts of Contexts on Repository-Level Code Generation [5.641402231731082]
  We present RepoExec, a novel benchmark designed to evaluate repository-level code generation.<n>We focus on three key aspects: executability, functional correctness through comprehensive test case generation, and accurate utilization of cross-file contexts.
  arXiv  Detail & Related papers  (2024-06-17T10:45:22Z)
• On The Importance of Reasoning for Context Retrieval in Repository-Level Code Editing [82.96523584351314]
  We decouple the task of context retrieval from the other components of the repository-level code editing pipelines. We conclude that while the reasoning helps to improve the precision of the gathered context, it still lacks the ability to identify its sufficiency.
  arXiv  Detail & Related papers  (2024-06-06T19:44:17Z)
• RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation [96.75695811963242]
  RepoCoder is a framework to streamline the repository-level code completion process. It incorporates a similarity-based retriever and a pre-trained code language model. It consistently outperforms the vanilla retrieval-augmented code completion approach.
  arXiv  Detail & Related papers  (2023-03-22T13:54:46Z)
• Deconstructing Self-Supervised Monocular Reconstruction: The Design Decisions that Matter [63.5550818034739]
  This paper presents a framework to evaluate state-of-the-art contributions to self-supervised monocular depth estimation. It includes pretraining, backbone, architectural design choices and loss functions. We re-implement, validate and re-evaluate 16 state-of-the-art contributions and introduce a new dataset.
  arXiv  Detail & Related papers  (2022-08-02T14:38:53Z)
• [not Rp] Reproducibility of 'Poincare dodecahedral space parameter estimates' [0.0]
  This paper attempts to reproduce a cosmic topology observational paper published in 2008. The reproduction steps are defined formally in a free-licensed git repository package "0807.4260"
  arXiv  Detail & Related papers  (2020-07-23T21:56:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.