Replications, Revisions, and Reanalyses: Managing Empirical Evidence in Software Engineering

• URL: http://arxiv.org/abs/2609.05983v1
• Date: Sat, 05 Sep 2026 08:57:53 GMT
• Title: Replications, Revisions, and Reanalyses: Managing Empirical Evidence in Software Engineering
• Abstract summary: validity of variance theories stems from the synthesis of multiple pieces of evidence.<n>Research synthesis in SE is rare and if done mostly limited to purely narrative syntheses.<n>We apply method engineering to construct a framework for research synthesis from proven, individual method fragments.
• Score: 6.147136878250444
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: One aspired outcome of empirical research on quantitative data is a variance theory, i.e., a quantification of the effect of an independent on a dependent variables. The validity of variance theories stems from the synthesis of multiple pieces of evidence, which increases its validity beyond the findings of a single study. However, research synthesis in SE is rare and if done mostly limited to purely narrative syntheses. At best, researchers perform meta-analyses to synthesize variance theories from several quantitative results. But even meta-analyses only produce reliable results when synthesizing exact replications yet fail to generalize from variations. We aim to extend the frontier of research synthesis beyond the state-of-the-art to systematically manage empirical evidence and its evolution. We apply method engineering to construct a framework for research synthesis from proven, individual method fragments. The framework allows researchers to put new evidence in a clear relation to an existing body of evidence and systematically expand knowledge about a studied phenomenon. We demonstrate the application of this framework to two fields of research by explicitly modeling the relationship between existing pieces of evidence. The framework puts three types of evolution of evidence into relation: (1) replications investigate the same hypothesis in a new context to improve external validity, (2) revisions challenge an existing hypothesis to improve internal validity, and (3) reanalyses replace analysis methods to improve conclusion validity. Through a systematic evolution of evidence and clear assessment criteria for each dimension of validity, the proposed framework can determine the frontier of a field of research. The framework provides a perspective to systematically evolve empirical evidence in SE, supporting more constructive and productive advances in our field.

Related papers

• EvoSCM: Scientific Belief Revision Through Causal Model Evolution and Experimentation [68.80901568810616]
  EvoSCM aims to equip scientific agents with explicit structural causal models that evolve as new evidence is collected.<n>We evaluate EvoSCM onPhysics, a benchmark requiring agents to uncover the hidden dynamics of noncanonical physical worlds through experimentation.
  arXiv  Detail & Related papers  (2026-09-01T16:55:56Z)
• SPARK: Skeleton-Guided Reasoning Synthesis from Large-Scale Scientific Literature [26.176405135367038]
  We introduce SPARK, a paper-oriented synthesis framework built on Sci-Base.<n> SPARK distills each paper into a compact reasoning skeleton capturing its central claims and supporting evidence.<n>We construct Spark-234K, a scientific reasoning dataset with substantially higher difficulty and diversity than existing resources.
  arXiv  Detail & Related papers  (2026-08-31T03:55:30Z)
• HiEviDR-Bench: A Benchmark for Hierarchical Evidence Aggregation in Deep Research [52.87128503345243]
  We introduce HiEviDR-Bench, a benchmark for evaluating Hierarchical Evidence Aggregation in Deep Research.<n>HiEviDR-Bench represents each instance with an explicit evidence graph that captures evidence selection, cross-source linking, and aggregation.<n>We develop a traceability-oriented evaluation framework with five dimensions: report quality, evidence traceability, citation accuracy, claim verification, and answer correctness.
  arXiv  Detail & Related papers  (2026-07-27T23:50:46Z)
• Externalizing Research Synthesis and Validation in AI Scientists through a Research Harness [22.631467433443362]
  Xcientist is a research harness that externalizes research synthesis and experimental validation into inspectable, contract-governed processes.<n>Xcientist literature organizes evidence, idea states, implementation plans, ablation records and repair traces as persistent research artifacts.
  arXiv  Detail & Related papers  (2026-06-17T09:52:14Z)
• PaperMind: Benchmarking Agentic Reasoning and Critique over Scientific Papers in Multimodal LLMs [68.27437550335709]
  We introduce PaperMind, a benchmark designed to evaluate integrated and agent-oriented scientific reasoning over research papers.<n>PaperMind is constructed from real scientific papers across seven domains, including agriculture, biology, chemistry, computer science, medicine, physics, and economics.<n>By analyzing model behavior across multiple tasks, PaperMind enables a diagnostic evaluation of integrated scientific reasoning behaviors.
  arXiv  Detail & Related papers  (2026-04-23T05:42:39Z)
• Exploring the Garden of Forking Paths in Empirical Software Engineering Research: A Multiverse Analysis [3.6324565773746147]
  We conduct a so-called multiverse analysis on a published empirical SE paper.<n>We identify nine pivotal analytical decisions with at least one equally defensible alternative.<n>The overwhelming majority produced qualitatively different, and sometimes even opposite, findings.
  arXiv  Detail & Related papers  (2025-12-09T18:47:00Z)
• SciRAG: Adaptive, Citation-Aware, and Outline-Guided Retrieval and Synthesis for Scientific Literature [52.36039386997026]
  We introduce SciRAG, an open-source framework for scientific literature exploration.<n>We introduce three key innovations: (1) adaptive retrieval that flexibly alternates between sequential and parallel evidence gathering; (2) citation-aware symbolic reasoning that leverages citation graphs to organize and filter documents; and (3) outline-guided synthesis that plans, critiques, and refines answers to ensure coherence and transparent attribution.
  arXiv  Detail & Related papers  (2025-11-18T11:09:19Z)
• Replications, Revisions, and Reanalyses: Managing Variance Theories in Software Engineering [4.147594239309427]
  Variance theories quantify the variance that one or more independent variables cause in a dependent variable.<n>In software engineering (SE), variance theories are used to quantify -- among others -- the impact of tools, techniques, and other treatments on software development outcomes.
  arXiv  Detail & Related papers  (2024-12-17T07:56:18Z)
• Causal Representation Learning from Multimodal Biomedical Observations [57.00712157758845]
  We develop flexible identification conditions for multimodal data and principled methods to facilitate the understanding of biomedical datasets.<n>Key theoretical contribution is the structural sparsity of causal connections between modalities.<n>Results on a real-world human phenotype dataset are consistent with established biomedical research.
  arXiv  Detail & Related papers  (2024-11-10T16:40:27Z)
• Retrieve to Explain: Evidence-driven Predictions for Explainable Drug Target Identification [0.791663505497707]
  Retrieve to Explain (R2E) is a retrieval-based model that scores and ranks all possible answers to a research question.<n>R2E represents each answer only in terms of its supporting evidence, with the answer itself masked.<n>We developed R2E for the challenging scientific discovery task of drug target identification.
  arXiv  Detail & Related papers  (2024-02-06T15:13:17Z)
• A Causal Framework for Decomposing Spurious Variations [68.12191782657437]
  We develop tools for decomposing spurious variations in Markovian and Semi-Markovian models. We prove the first results that allow a non-parametric decomposition of spurious effects. The described approach has several applications, ranging from explainable and fair AI to questions in epidemiology and medicine.
  arXiv  Detail & Related papers  (2023-06-08T09:40:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.