Content and structure of laboratory packages for software engineering experiments

• URL: http://arxiv.org/abs/2402.07217v1
• Date: Sun, 11 Feb 2024 14:29:15 GMT
• Title: Content and structure of laboratory packages for software engineering experiments
• Authors: Mart\'in Solari, Sira Vegas, Natalia Juristo
• Abstract summary: This paper investigates the experiment replication process to find out what information is needed to successfully replicate an experiment. Our objective is to propose the content and structure of laboratory packages for software engineering experiments.
• Score: 1.3584003182788122
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Context: Experiment replications play a central role in the scientific method. Although software engineering experimentation has matured a great deal, the number of experiment replications is still relatively small. Software engineering experiments are composed of complex concepts, procedures and artefacts. Laboratory packages are a means of transfer-ring knowledge among researchers to facilitate experiment replications. Objective: This paper investigates the experiment replication process to find out what information is needed to successfully replicate an experiment. Our objective is to propose the content and structure of laboratory packages for software engineering experiments. Method: We evaluated seven replications of three different families of experiments. Each replication had a different experimenter who was, at the time, unfamiliar with the experi-ment. During the first iterations of the study, we identified experimental incidents and then proposed a laboratory package structure that addressed these incidents, including docu-ment usability improvements. We used the later iterations to validate and generalize the laboratory package structure for use in all software engineering experiments. We aimed to solve a specific problem, while at the same time looking at how to contribute to the body of knowledge on laboratory packages. Results: We generated a laboratory package for three different experiments. These packages eased the replication of the respective experiments. The evaluation that we conducted shows that the laboratory package proposal is acceptable and reduces the effort currently required to replicate experiments in software engineering. Conclusion: We think that the content and structure that we propose for laboratory pack-ages can be useful for other software engineering experiments.

Related papers

• An instructional lab apparatus for quantum experiments with single nitrogen-vacancy centers in diamond [0.14993626998062629]
  We present an experimental apparatus for performing quantum experiments with single nitrogen-vacancy (NV) centers in diamond. We describe the basic physics of the NV center and give examples of potential experiments that can be performed with this apparatus. The apparatus described here enables students to write their own experimental control and data analysis software from scratch all within a single semester of a typical lab course.
  arXiv  Detail & Related papers  (2024-07-22T16:10:59Z)
• DISCOVERYWORLD: A Virtual Environment for Developing and Evaluating Automated Scientific Discovery Agents [49.74065769505137]
  We introduce DISCOVERYWORLD, the first virtual environment for developing and benchmarking an agent's ability to perform complete cycles of novel scientific discovery. It includes 120 different challenge tasks spanning eight topics each with three levels of difficulty and several parametric variations. We find that strong baseline agents, that perform well in prior published environments, struggle on most DISCOVERYWORLD tasks.
  arXiv  Detail & Related papers  (2024-06-10T20:08:44Z)
• Quantum software experiments: A reporting and laboratory package structure guidelines [1.2701060255765317]
  No standardized guidelines exist for describing experiments or outlining the necessary structures for quantum software laboratory packages. This paper endeavors to enhance the replicability and verifiability of quantum software experiments.
  arXiv  Detail & Related papers  (2024-05-07T11:05:47Z)
• MLXP: A Framework for Conducting Replicable Experiments in Python [63.37350735954699]
  We propose MLXP, an open-source, simple, and lightweight experiment management tool based on Python. It streamlines the experimental process with minimal overhead while ensuring a high level of practitioner overhead.
  arXiv  Detail & Related papers  (2024-02-21T14:22:20Z)
• A Backend Platform for Supporting the Reproducibility of Computational Experiments [2.1485350418225244]
  It is challenging to recreate the same environment using the same frameworks, code, data sources, programming languages, dependencies, and so on. In this work, we propose an Integrated Development Environment allowing the share, configuration, packaging and execution of an experiment. We have been able to successfully reproduce 20 (80%) of these experiments achieving the results reported in such works with minimum effort.
  arXiv  Detail & Related papers  (2023-06-29T10:29:11Z)
• GFlowNets for AI-Driven Scientific Discovery [74.27219800878304]
  We present a new probabilistic machine learning framework called GFlowNets. GFlowNets can be applied in the modeling, hypotheses generation and experimental design stages of the experimental science loop. We argue that GFlowNets can become a valuable tool for AI-driven scientific discovery.
  arXiv  Detail & Related papers  (2023-02-01T17:29:43Z)
• PyExperimenter: Easily distribute experiments and track results [63.871474825689134]
  PyExperimenter is a tool to facilitate the setup, documentation, execution, and subsequent evaluation of results from an empirical study of algorithms. It is intended to be used by researchers in the field of artificial intelligence, but is not limited to those.
  arXiv  Detail & Related papers  (2023-01-16T10:43:02Z)
• Learning One Abstract Bit at a Time Through Self-Invented Experiments Encoded as Neural Networks [8.594140167290098]
  We present an empirical analysis of the automatic generation of interesting experiments. In the first setting, we investigate self-invented experiments in a reinforcement-providing environment. In the second setting, pure thought experiments are implemented as the weights of recurrent neural networks.
  arXiv  Detail & Related papers  (2022-12-29T17:11:49Z)
• Active Exploration via Experiment Design in Markov Chains [86.41407938210193]
  A key challenge in science and engineering is to design experiments to learn about some unknown quantity of interest. We propose an algorithm that efficiently selects policies whose measurement allocation converges to the optimal one. In addition to our theoretical analysis, we showcase our framework on applications in ecological surveillance and pharmacology.
  arXiv  Detail & Related papers  (2022-06-29T00:04:40Z)
• Experiments as Code: A Concept for Reproducible, Auditable, Debuggable, Reusable, & Scalable Experiments [7.557948558412152]
  A common concern in experimental research is the auditability and of experiments. We propose the "Experiments as Code" paradigm, where the whole experiment is not only documented but additionally the automation code is provided.
  arXiv  Detail & Related papers  (2022-02-24T12:15:00Z)
• Federated Learning of Molecular Properties in a Heterogeneous Setting [79.00211946597845]
  We introduce federated heterogeneous molecular learning to address these challenges. Federated learning allows end-users to build a global model collaboratively while preserving the training data distributed over isolated clients. FedChem should enable a new type of collaboration for improving AI in chemistry that mitigates concerns about valuable chemical data.
  arXiv  Detail & Related papers  (2021-09-15T12:49:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.