Rethinking Software Empirical Studies with Structural Causal Models

• URL: http://arxiv.org/abs/2605.28482v1
• Date: Wed, 27 May 2026 13:41:05 GMT
• Title: Rethinking Software Empirical Studies with Structural Causal Models
• Abstract summary: Causal Inference offers a fundamental approach for advancing empirical software engineering (ESE) beyond traditional statistical association.<n>This paper introduces CausalSE, a framework that operationalizes Judea Pearl's causal inference paradigm in ESE context.
• Score: 12.18241004961061
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Causal Inference offers a fundamental approach for advancing empirical software engineering (ESE) beyond traditional statistical association, enabling researchers to rigorously identify and quantify causal relationships in software experiments. This paper introduces CausalSE, a framework that operationalizes Judea Pearl's causal inference paradigm in ESE context. The paper focuses on Structural Causal Models (SCMs) to address the limitations of classical statistical methods in mitigating confounding bias. Through a case study using the Galeras dataset and propensity score matching, we demonstrate how CausalSE disentangles the effect of prompt engineering strategies on code generation outcomes in a popular LLM (i.e., GPT-3). The results reveal that while associational analyses can suggest improvements in certain interventions (e.g., more complex prompts), causal analysis often does not find a significant treatment effect, highlighting the risk of false positives when confounding is not addressed. By providing a tutorial-based methodology and a real-world case study, this work equips software researchers with practical tools to design, analyze, and interpret software experiments with methodological rigor, ultimately enabling more informed and actionable conclusions in both research and practice.

Related papers

• ECLAIR: A Causally-Grounded AI Framework for Scientific Discovery in Empirical Software Engineering [51.19819833971252]
  This paper introduces ECLAIR, a causally grounded AI framework that integrates Large Language Models into every stage of the scientific process.<n>ECLAIR treats LLMs as active scientific agents operating under the principles of causal inference.
  arXiv  Detail & Related papers  (2026-08-03T14:47:03Z)
• CDFM: Towards a General-Purpose Causal Discovery Foundation Model [62.32138962508216]
  Causal discovery is the process of recovering underlying causal structures from observational data.<n>We formulate the Causal Discovery Foundation Model (CDFM) as a unified, general-purpose framework for zero-shot structural inference.<n>CDFM consistently outperforms traditional algorithms, driving a paradigm shift toward a general-purpose causal discovery foundation model.
  arXiv  Detail & Related papers  (2026-07-13T12:58:28Z)
• Causal Ensemble Agent: Hierarchical Causal Discovery with LLM-guided Expert Reweighting [95.24017293537025]
  Causal Ensemble Agent (CEA) is a novel framework that aggregates structural insights from statistical discovery experts across different graph levels.<n>CEA achieves the strongest overall performance across a wide range of causal discovery methods.
  arXiv  Detail & Related papers  (2026-06-09T09:09:07Z)
• Exploring the Garden of Forking Paths in Empirical Software Engineering Research: A Multiverse Analysis [3.6324565773746147]
  We conduct a so-called multiverse analysis on a published empirical SE paper.<n>We identify nine pivotal analytical decisions with at least one equally defensible alternative.<n>The overwhelming majority produced qualitatively different, and sometimes even opposite, findings.
  arXiv  Detail & Related papers  (2025-12-09T18:47:00Z)
• Data Fusion for Partial Identification of Causal Effects [62.56890808004615]
  We propose a novel partial identification framework that enables researchers to answer key questions.<n>Is the causal effect positive or negative? and How severe must assumption violations be to overturn this conclusion?<n>We apply our framework to the Project STAR study, which investigates the effect of classroom size on students' third-grade standardized test performance.
  arXiv  Detail & Related papers  (2025-05-30T07:13:01Z)
• Causal-Copilot: An Autonomous Causal Analysis Agent [25.5944359329613]
  Causal-Copilot is an autonomous agent that operationalizes expert-level causal analysis.<n>It supports interactive refinement through natural language, lowering the barrier for non-specialists.<n>Our system fosters a virtuous cycle -- expanding access to advanced causal methods for domain experts.
  arXiv  Detail & Related papers  (2025-04-17T18:05:39Z)
• The Curse of CoT: On the Limitations of Chain-of-Thought in In-Context Learning [56.574829311863446]
  Chain-of-Thought (CoT) prompting has been widely recognized for its ability to enhance reasoning capabilities in large language models (LLMs)<n>We demonstrate that CoT and its reasoning variants consistently underperform direct answering across varying model scales and benchmark complexities.<n>Our analysis uncovers a fundamental hybrid mechanism of explicit-implicit reasoning driving CoT's performance in pattern-based ICL.
  arXiv  Detail & Related papers  (2025-04-07T13:51:06Z)
• Causality can systematically address the monsters under the bench(marks) [64.36592889550431]
  Benchmarks are plagued by various biases, artifacts, or leakage.<n>Models may behave unreliably due to poorly explored failure modes.<n> causality offers an ideal framework to systematically address these challenges.
  arXiv  Detail & Related papers  (2025-02-07T17:01:37Z)
• Failure Modes of LLMs for Causal Reasoning on Narratives [51.19592551510628]
  We investigate the interaction between world knowledge and logical reasoning.<n>We find that state-of-the-art large language models (LLMs) often rely on superficial generalizations.<n>We show that simple reformulations of the task can elicit more robust reasoning behavior.
  arXiv  Detail & Related papers  (2024-10-31T12:48:58Z)
• Revisiting Spurious Correlation in Domain Generalization [12.745076668687748]
  We build a structural causal model (SCM) to describe the causality within data generation process. We further conduct a thorough analysis of the mechanisms underlying spurious correlation. In this regard, we propose to control confounding bias in OOD generalization by introducing a propensity score weighted estimator.
  arXiv  Detail & Related papers  (2024-06-17T13:22:00Z)
• A Second Look at the Impact of Passive Voice Requirements on Domain Modeling: Bayesian Reanalysis of an Experiment [4.649794383775257]
  We reanalyze the only known controlled experiment investigating the impact of passive voice on the subsequent activity of domain modeling. Our results reveal that the effects observed by the original authors turned out to be much less significant than previously assumed.
  arXiv  Detail & Related papers  (2024-02-16T16:24:00Z)
• Advancing Counterfactual Inference through Nonlinear Quantile Regression [77.28323341329461]
  We propose a framework for efficient and effective counterfactual inference implemented with neural networks. The proposed approach enhances the capacity to generalize estimated counterfactual outcomes to unseen data. Empirical results conducted on multiple datasets offer compelling support for our theoretical assertions.
  arXiv  Detail & Related papers  (2023-06-09T08:30:51Z)
• Towards Causal Analysis of Empirical Software Engineering Data: The Impact of Programming Languages on Coding Competitions [10.51554436183424]
  This paper discusses some novel techniques based on structural causal models. We apply these ideas to analyzing public data about programmer performance in Code Jam. We find considerable differences between a purely associational and a causal analysis of the very same data.
  arXiv  Detail & Related papers  (2023-01-18T13:46:16Z)
• A Survey on Causal Inference [64.45536158710014]
  Causal inference is a critical research topic across many domains, such as statistics, computer science, education, public policy and economics. Various causal effect estimation methods for observational data have sprung up.
  arXiv  Detail & Related papers  (2020-02-05T21:35:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.