AI-Assisted Computational Reproducibility on the FABRIC Testbed

• URL: http://arxiv.org/abs/2606.25879v1
• Date: Wed, 24 Jun 2026 14:23:14 GMT
• Title: AI-Assisted Computational Reproducibility on the FABRIC Testbed
• Abstract summary: We show how the FABRIC testbed, combined with large language model (LLM) coding assistants through LoomAI, can simplify reproducing experiments across multiple domains.<n>We reproduced three case studies on FABRIC, covering BBR-family congestion-control evaluations, LAMMPS molecular dynamics scaling benchmarks on a CPU-only MPI cluster, and stress protein homeostasis genomics pipelines.<n>Across the case studies, the AI-assisted workflow reduced reproduction effort by roughly 4--6 times.
• Score: 2.756713570455716
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Computational reproducibility remains difficult despite being central to scientific research. In this paper, we show how the international FABRIC testbed, combined with large language model (LLM) coding assistants through LoomAI, can simplify reproducing published experiments across multiple domains. We reproduced three case studies on FABRIC, covering BBR-family congestion-control evaluations, LAMMPS molecular dynamics scaling benchmarks on a CPU-only MPI cluster, and stress protein homeostasis genomics pipelines. Rather than focusing only on matching numerical outputs, we evaluate whether the reproduced experiments support the same scientific conclusions as the original studies. The AI assistant was effective in setting up the environment, adapting code, and debugging, but struggled with the analysis stages that lacked clearly defined workflows, which required human guidance to establish execution order and data dependencies. Across the case studies, the AI-assisted workflow reduced reproduction effort by roughly 4--6 times. We conclude with practical recommendations for improving AI-assisted reproducibility on research testbeds.

Related papers

• Claw AI Lab: An Autonomous Multi-Agent Research Team [73.07229229907435]
  We present Claw AI Lab, a lab-native autonomous research platform.<n>Rather than centering the system around a single agent or a fixed serial workflow, we allow users to instantiate a full research team from one prompt.<n>Claw-Code Harness connects locals, datasets, and checkpoints to runnable experiments and feeds execution artifacts back into the research loop.
  arXiv  Detail & Related papers  (2026-05-21T16:02:53Z)
• Can Coding Agents Reproduce Findings in Computational Materials Science? [49.254975563645786]
  We present AutoMat, a benchmark for evaluating large language models' ability to reproduce scientific claims.<n>By working closely with subject matter experts, we curate a set of claims from real materials science papers to test whether coding agents can recover and execute the end-to-end workflow.<n>Results show that current LLM-based agents obtain low overall success rates on AutoMat, with the best-performing setting achieving a success rate of only 54.1%.
  arXiv  Detail & Related papers  (2026-05-01T17:42:12Z)
• HeurekaBench: A Benchmarking Framework for AI Co-scientist [2.206319727896241]
  HeurekaBench is a framework to create benchmarks with exploratory, open-ended research questions for experimental datasets.<n>We instantiate the framework in single-cell biology to obtain sc-HeurekaBench benchmark and use it to compare state-of-the-art single-cell agents.<n>We find that the addition of a critic module can improve ill-formed responses for open-source LLM-based agents by up to 22% and close the gap with their closed-source counterparts.
  arXiv  Detail & Related papers  (2026-01-04T22:16:42Z)
• SelfAI: Building a Self-Training AI System with LLM Agents [79.10991818561907]
  SelfAI is a general multi-agent platform that combines a User Agent for translating high-level research objectives into standardized experimental configurations.<n>An Experiment Manager orchestrates parallel, fault-tolerant training across heterogeneous hardware while maintaining a structured knowledge base for continuous feedback.<n>Across regression, computer vision, scientific computing, medical imaging, and drug discovery benchmarks, SelfAI consistently achieves strong performance and reduces redundant trials.
  arXiv  Detail & Related papers  (2025-11-29T09:18:39Z)
• AblationBench: Evaluating Automated Planning of Ablations in Empirical AI Research [34.173947968362675]
  AblationBench is a benchmark suite for evaluating agents on ablation planning tasks in empirical AI research.<n>It includes two tasks: AuthorAblation, which helps authors propose ablation experiments based on a method section, and ReviewerAblation, which helps reviewers find missing ablations in a full paper.<n>For both tasks, we develop LM-based judges that serve as an automatic evaluation framework.
  arXiv  Detail & Related papers  (2025-07-09T12:07:38Z)
• AutoReproduce: Automatic AI Experiment Reproduction with Paper Lineage [62.049868205196425]
  AutoReproduce is a framework capable of automatically reproducing experiments described in research papers in an end-to-end manner.<n>Results show that AutoReproduce achieves an average performance gap of $22.1%$ on $89.74%$ of the executable experiment runs.
  arXiv  Detail & Related papers  (2025-05-27T03:15:21Z)
• Autonomous Microscopy Experiments through Large Language Model Agents [4.241267255764773]
  Large language models (LLMs) are revolutionizing self driving laboratories (SDLs) for materials research.<n>We introduce Artificially Intelligent Lab Assistant (AILA), a framework automating atomic force microscopy through LLM driven agents.<n>We find that state of the art models struggle with basic tasks and coordination scenarios.
  arXiv  Detail & Related papers  (2024-12-18T09:35:28Z)
• MLXP: A Framework for Conducting Replicable Experiments in Python [63.37350735954699]
  We propose MLXP, an open-source, simple, and lightweight experiment management tool based on Python. It streamlines the experimental process with minimal overhead while ensuring a high level of practitioner overhead.
  arXiv  Detail & Related papers  (2024-02-21T14:22:20Z)
• The NCI Imaging Data Commons as a platform for reproducible research in computational pathology [0.0773931605896092]
  Reproducibility is a major challenge in developing machine learning (ML)-based solutions in computational pathology (CompPath) The NCI Imaging Data Commons (IDC) provides >120 cancer image collections according to the FAIR principles and is designed to be used with cloud ML services. We implement two experiments in which a representative ML-based method for classifying lung tumor tissue was trained and/or evaluated on different datasets.
  arXiv  Detail & Related papers  (2023-03-16T14:32:50Z)
• Deep Bayesian Active Learning for Accelerating Stochastic Simulation [74.58219903138301]
  Interactive Neural Process (INP) is a deep active learning framework for simulations and with active learning approaches. For active learning, we propose a novel acquisition function, Latent Information Gain (LIG), calculated in the latent space of NP based models. The results demonstrate STNP outperforms the baselines in the learning setting and LIG achieves the state-of-the-art for active learning.
  arXiv  Detail & Related papers  (2021-06-05T01:31:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.