A Framework for Supporting the Reproducibility of Computational Experiments in Multiple Scientific Domains

• URL: http://arxiv.org/abs/2503.07080v3
• Date: Thu, 13 Mar 2025 14:40:30 GMT
• Title: A Framework for Supporting the Reproducibility of Computational Experiments in Multiple Scientific Domains
• Authors: Lázaro Costa, Susana Barbosa, Jácome Cunha,
• Abstract summary: In recent years, the research community, but also the general public, has raised serious questions about the replicability of scientific work.<n>We propose a framework, known as SciRep, that supports the configuration, execution, and packaging of computational experiments.<n>Our approach allows the creation of a package for experiments from multiple scientific fields, which can be re-executed on any computer.
• Score: 2.147712260420443
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In recent years, the research community, but also the general public, has raised serious questions about the reproducibility and replicability of scientific work. Since many studies include some kind of computational work, these issues are also a technological challenge, not only in computer science, but also in most research domains. Computational replicability and reproducibility are not easy to achieve due to the variety of computational environments that can be used. Indeed, it is challenging to recreate the same environment via the same frameworks, code, programming languages, dependencies, and so on. We propose a framework, known as SciRep, that supports the configuration, execution, and packaging of computational experiments by defining their code, data, programming languages, dependencies, databases, and commands to be executed. After the initial configuration, the experiments can be executed any number of times, always producing exactly the same results. Our approach allows the creation of a reproducibility package for experiments from multiple scientific fields, from medicine to computer science, which can be re-executed on any computer. The produced package acts as a capsule, holding absolutely everything necessary to re-execute the experiment. To evaluate our framework, we compare it with three state-of-the-art tools and use it to reproduce 18 experiments extracted from published scientific articles. With our approach, we were able to execute 16 (89%) of those experiments, while the others reached only 61%, thus showing that our approach is effective. Moreover, all the experiments that were executed produced the results presented in the original publication. Thus, SciRep was able to reproduce 100% of the experiments it could run.

Related papers

• A Dataset For Computational Reproducibility [2.147712260420443]
  This article introduces a dataset of computational experiments covering a broad spectrum of scientific fields. It incorporates details about software dependencies, execution steps, and configurations necessary for accurate reproduction. It provides a universal benchmark by establishing a standardized dataset for objectively evaluating and comparing the effectiveness of tools.
  arXiv  Detail & Related papers  (2025-04-11T16:45:10Z)
• CodeScientist: End-to-End Semi-Automated Scientific Discovery with Code-based Experimentation [48.12054700748627]
  We introduce CodeScientist, a novel ASD system that frames ideation and experiment construction as a form of genetic search jointly. We use this paradigm to conduct hundreds of automated experiments on machine-generated ideas broadly in the domain of agents and virtual environments.
  arXiv  Detail & Related papers  (2025-03-20T22:37:17Z)
• Many Heads Are Better Than One: Improved Scientific Idea Generation by A LLM-Based Multi-Agent System [62.832818186789545]
  Virtual Scientists (VirSci) is a multi-agent system designed to mimic the teamwork inherent in scientific research.<n>VirSci organizes a team of agents to collaboratively generate, evaluate, and refine research ideas.<n>We show that this multi-agent approach outperforms the state-of-the-art method in producing novel scientific ideas.
  arXiv  Detail & Related papers  (2024-10-12T07:16:22Z)
• DISCOVERYWORLD: A Virtual Environment for Developing and Evaluating Automated Scientific Discovery Agents [49.74065769505137]
  We introduce DISCOVERYWORLD, the first virtual environment for developing and benchmarking an agent's ability to perform complete cycles of novel scientific discovery. It includes 120 different challenge tasks spanning eight topics each with three levels of difficulty and several parametric variations. We find that strong baseline agents, that perform well in prior published environments, struggle on most DISCOVERYWORLD tasks.
  arXiv  Detail & Related papers  (2024-06-10T20:08:44Z)
• MLXP: A Framework for Conducting Replicable Experiments in Python [63.37350735954699]
  We propose MLXP, an open-source, simple, and lightweight experiment management tool based on Python. It streamlines the experimental process with minimal overhead while ensuring a high level of practitioner overhead.
  arXiv  Detail & Related papers  (2024-02-21T14:22:20Z)
• Content and structure of laboratory packages for software engineering experiments [1.3584003182788122]
  This paper investigates the experiment replication process to find out what information is needed to successfully replicate an experiment. Our objective is to propose the content and structure of laboratory packages for software engineering experiments.
  arXiv  Detail & Related papers  (2024-02-11T14:29:15Z)
• A Backend Platform for Supporting the Reproducibility of Computational Experiments [2.1485350418225244]
  It is challenging to recreate the same environment using the same frameworks, code, data sources, programming languages, dependencies, and so on. In this work, we propose an Integrated Development Environment allowing the share, configuration, packaging and execution of an experiment. We have been able to successfully reproduce 20 (80%) of these experiments achieving the results reported in such works with minimum effort.
  arXiv  Detail & Related papers  (2023-06-29T10:29:11Z)
• GFlowNets for AI-Driven Scientific Discovery [74.27219800878304]
  We present a new probabilistic machine learning framework called GFlowNets. GFlowNets can be applied in the modeling, hypotheses generation and experimental design stages of the experimental science loop. We argue that GFlowNets can become a valuable tool for AI-driven scientific discovery.
  arXiv  Detail & Related papers  (2023-02-01T17:29:43Z)
• Caching and Reproducibility: Making Data Science experiments faster and FAIRer [25.91002326340444]
  Small to medium-scale data science experiments often rely on research software developed ad-hoc by individual scientists or small teams. We suggest making caching an integral part of the research software development process, even before the first line of code is written.
  arXiv  Detail & Related papers  (2022-11-08T07:11:02Z)
• SIERRA: A Modular Framework for Research Automation and Reproducibility [6.1678491628787455]
  We present SIERRA, a novel framework for accelerating research development and improving results. SIERRA accelerates research by automating the process of generating executable experiments from queries over independent variables. It employs a modular architecture enabling easy customization and extension for the needs of individual researchers.
  arXiv  Detail & Related papers  (2022-08-16T15:36:34Z)
• Experiments as Code: A Concept for Reproducible, Auditable, Debuggable, Reusable, & Scalable Experiments [7.557948558412152]
  A common concern in experimental research is the auditability and of experiments. We propose the "Experiments as Code" paradigm, where the whole experiment is not only documented but additionally the automation code is provided.
  arXiv  Detail & Related papers  (2022-02-24T12:15:00Z)
• Dynamic Causal Effects Evaluation in A/B Testing with a Reinforcement Learning Framework [68.96770035057716]
  A/B testing is a business strategy to compare a new product with an old one in pharmaceutical, technological, and traditional industries. This paper introduces a reinforcement learning framework for carrying A/B testing in online experiments.
  arXiv  Detail & Related papers  (2020-02-05T10:25:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.