Towards Improving the External Validity of Software Engineering Experiments with Transportability Methods

• URL: http://arxiv.org/abs/2604.08200v1
• Date: Thu, 09 Apr 2026 12:57:26 GMT
• Title: Towards Improving the External Validity of Software Engineering Experiments with Transportability Methods
• Abstract summary: We introduce transportability methods, their prerequisites, and demonstrate their potential through a simulation.<n>We then outline several SE research scenarios in which these methods could apply, e.g., how to effectively use students as substitutes for developers.<n>Finally, we outline a road map and practical guidelines to support SE researchers in applying them.
• Score: 3.9236728810856714
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Controlled experiments are a core research method in software engineering (SE) for validating causal claims. However, recruiting a sample of participants that represents the intended target population is often difficult or expensive, which limits the external validity of experimental results. At the same time, SE researchers often have access to much larger amounts of observational than experimental data (e.g., from repositories, issue trackers, logs, surveys and industrial processes). Transportability methods combine these data from experimental and observational studies to "transport" results from the experimental sample to a broader, more representative sample of the target population. Although the ability to combine observational and experimental data in a principled way could substantially benefit empirical SE research, transportability methods have - to our knowledge - not been adopted in SE. In this vision, we aim to help make that adoption possible. To that end, we introduce transportability methods, their prerequisites, and demonstrate their potential through a simulation. We then outline several SE research scenarios in which these methods could apply, e.g., how to effectively use students as substitutes for developers. Finally, we outline a road map and practical guidelines to support SE researchers in applying them. Adopting transportability methods in SE research can strengthen the external validity of controlled experiments and help the field produce results that are both more reliable and more useful in practice.

Related papers

• Learning plug-in surrogate endpoints for randomized experiments [48.46651943268683]
  Surrogate endpoints are used in place of long-term outcomes in randomized experiments.<n>A short-term surrogate is good if the result of an experiment using the surrogate is predictive of the result of a hypothetical study using the real outcome.<n>We propose two methods for learning plug-in surrogates that maximize effect predictiveness.
  arXiv  Detail & Related papers  (2026-05-12T12:31:04Z)
• Tutorial Debriefing: Applied Statistical Causal Inference in Requirements Engineering [4.29699238971962]
  The software engineering (SE) research community strives to contribute to the betterment of the target population of our research: software producers and consumers.<n>We will only achieve this betterment if we manage to transfer the knowledge acquired during research into practice.<n>The value of these contributions hinges on the assumption that applying them causes an improvement of the development process, user experience, or other performance metrics.
  arXiv  Detail & Related papers  (2025-11-05T21:43:53Z)
• MOOSE-Chem3: Toward Experiment-Guided Hypothesis Ranking via Simulated Experimental Feedback [136.27567671480156]
  We introduce experiment-guided ranking, which prioritizes hypotheses based on feedback from prior tests.<n>We frame experiment-guided ranking as a sequential decision-making problem.<n>Our approach significantly outperforms pre-experiment baselines and strong ablations.
  arXiv  Detail & Related papers  (2025-05-23T13:24:50Z)
• Simulating Field Experiments with Large Language Models [0.6144680854063939]
  This paper pioneers the utilization of large language models (LLMs) for simulating field experiments. By introducing two novel prompting strategies, observer and participant modes, we demonstrate the ability of LLMs to both predict outcomes and replicate participant responses within complex field settings. Our findings indicate a promising alignment with actual experimental results in certain scenarios, achieving a stimulation accuracy of 66% in observer mode.
  arXiv  Detail & Related papers  (2024-08-19T03:41:43Z)
• Adaptive Instrument Design for Indirect Experiments [48.815194906471405]
  Unlike RCTs, indirect experiments estimate treatment effects by leveragingconditional instrumental variables. In this paper we take the initial steps towards enhancing sample efficiency for indirect experiments by adaptively designing a data collection policy. Our main contribution is a practical computational procedure that utilizes influence functions to search for an optimal data collection policy.
  arXiv  Detail & Related papers  (2023-12-05T02:38:04Z)
• Opportunities for Adaptive Experiments to Enable Continuous Improvement in Computer Science Education [7.50867730317249]
  In adaptive experiments, data is analyzed and utilized as different conditions are deployed to students. These algorithms can then dynamically deploy the most effective conditions in subsequent interactions with students. This work paves the way for exploring the importance of adaptive experiments in bridging research and practice to achieve continuous improvement.
  arXiv  Detail & Related papers  (2023-10-18T20:54:59Z)
• A Double Machine Learning Approach to Combining Experimental and Observational Data [58.05402364136958]
  We propose a double machine learning approach to combine experimental and observational studies.<n>Our framework proposes a falsification test for external validity and ignorability under milder assumptions.
  arXiv  Detail & Related papers  (2023-07-04T02:53:11Z)
• GFlowNets for AI-Driven Scientific Discovery [74.27219800878304]
  We present a new probabilistic machine learning framework called GFlowNets. GFlowNets can be applied in the modeling, hypotheses generation and experimental design stages of the experimental science loop. We argue that GFlowNets can become a valuable tool for AI-driven scientific discovery.
  arXiv  Detail & Related papers  (2023-02-01T17:29:43Z)
• Using Adaptive Experiments to Rapidly Help Students [5.446351709118483]
  We evaluate the effect of homework email reminders in students by conducting an adaptive experiment using the Thompson Sampling algorithm. We raise a range of open questions about the conditions under which adaptive randomized experiments may be more or less useful.
  arXiv  Detail & Related papers  (2022-08-10T00:43:05Z)
• Active Exploration via Experiment Design in Markov Chains [86.41407938210193]
  A key challenge in science and engineering is to design experiments to learn about some unknown quantity of interest. We propose an algorithm that efficiently selects policies whose measurement allocation converges to the optimal one. In addition to our theoretical analysis, we showcase our framework on applications in ecological surveillance and pharmacology.
  arXiv  Detail & Related papers  (2022-06-29T00:04:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.