Testing Causality in Scientific Modelling Software
- URL: http://arxiv.org/abs/2209.00357v2
- Date: Fri, 30 Jun 2023 14:01:25 GMT
- Title: Testing Causality in Scientific Modelling Software
- Authors: Andrew G. Clark, Michael Foster, Benedikt Prifling, Neil Walkinshaw,
Robert M. Hierons, Volker Schmidt, Robert D. Turner
- Abstract summary: Causal Testing Framework is a framework that uses Causal Inference techniques to establish causal effects from existing data.
We present three case studies covering real-world scientific models, demonstrating how the Causal Testing Framework can infer metamorphic test outcomes.
- Score: 0.26388783516590225
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: From simulating galaxy formation to viral transmission in a pandemic,
scientific models play a pivotal role in developing scientific theories and
supporting government policy decisions that affect us all. Given these critical
applications, a poor modelling assumption or bug could have far-reaching
consequences. However, scientific models possess several properties that make
them notoriously difficult to test, including a complex input space, long
execution times, and non-determinism, rendering existing testing techniques
impractical. In fields such as epidemiology, where researchers seek answers to
challenging causal questions, a statistical methodology known as Causal
Inference has addressed similar problems, enabling the inference of causal
conclusions from noisy, biased, and sparse data instead of costly experiments.
This paper introduces the Causal Testing Framework: a framework that uses
Causal Inference techniques to establish causal effects from existing data,
enabling users to conduct software testing activities concerning the effect of
a change, such as Metamorphic Testing, a posteriori. We present three case
studies covering real-world scientific models, demonstrating how the Causal
Testing Framework can infer metamorphic test outcomes from reused, confounded
test data to provide an efficient solution for testing scientific modelling
software.
Related papers
- Smoke and Mirrors in Causal Downstream Tasks [59.90654397037007]
This paper looks at the causal inference task of treatment effect estimation.
We assume binary effects that are recorded as high-dimensional images in a Randomized Controlled Trial.
We compare 6 480 models fine-tuned from state-of-the-art visual backbones.
We find that the sampling and modeling choices significantly affect the accuracy of the causal estimate.
arXiv Detail & Related papers (2024-05-27T13:26:34Z) - The Causal Chambers: Real Physical Systems as a Testbed for AI Methodology [10.81691411087626]
In some fields of AI, machine learning and statistics, the validation of new methods and algorithms is often hindered by the scarcity of suitable real-world datasets.
We have constructed two devices that allow us to quickly and inexpensively produce large datasets from non-trivial but well-understood physical systems.
arXiv Detail & Related papers (2024-04-17T13:00:52Z) - A Second Look at the Impact of Passive Voice Requirements on Domain
Modeling: Bayesian Reanalysis of an Experiment [4.649794383775257]
We reanalyze the only known controlled experiment investigating the impact of passive voice on the subsequent activity of domain modeling.
Our results reveal that the effects observed by the original authors turned out to be much less significant than previously assumed.
arXiv Detail & Related papers (2024-02-16T16:24:00Z) - Targeted Reduction of Causal Models [55.11778726095353]
Causal Representation Learning offers a promising avenue to uncover interpretable causal patterns in simulations.
We introduce Targeted Causal Reduction (TCR), a method for condensing complex intervenable models into a concise set of causal factors.
Its ability to generate interpretable high-level explanations from complex models is demonstrated on toy and mechanical systems.
arXiv Detail & Related papers (2023-11-30T15:46:22Z) - Partially Specified Causal Simulations [0.0]
Many causal inference literature tend to design over-restricted or misspecified studies.
We introduce partially randomized causal simulation (PARCS), a simulation framework that meets those desiderata.
We reproduce and extend the simulation studies of two well-known causal discovery and missing data analysis papers.
arXiv Detail & Related papers (2023-09-19T10:50:35Z) - A Causal Framework for Decomposing Spurious Variations [68.12191782657437]
We develop tools for decomposing spurious variations in Markovian and Semi-Markovian models.
We prove the first results that allow a non-parametric decomposition of spurious effects.
The described approach has several applications, ranging from explainable and fair AI to questions in epidemiology and medicine.
arXiv Detail & Related papers (2023-06-08T09:40:28Z) - GFlowNets for AI-Driven Scientific Discovery [74.27219800878304]
We present a new probabilistic machine learning framework called GFlowNets.
GFlowNets can be applied in the modeling, hypotheses generation and experimental design stages of the experimental science loop.
We argue that GFlowNets can become a valuable tool for AI-driven scientific discovery.
arXiv Detail & Related papers (2023-02-01T17:29:43Z) - A Causal Framework to Quantify the Robustness of Mathematical Reasoning
with Language Models [81.15974174627785]
We study the behavior of language models in terms of robustness and sensitivity to direct interventions in the input space.
Our analysis shows that robustness does not appear to continuously improve as a function of size, but the GPT-3 Davinci models (175B) achieve a dramatic improvement in both robustness and sensitivity compared to all other GPT variants.
arXiv Detail & Related papers (2022-10-21T15:12:37Z) - Evaluating Causal Inference Methods [0.4588028371034407]
We introduce a deep generative model-based framework, Credence, to validate causal inference methods.
Our work introduces a deep generative model-based framework, Credence, to validate causal inference methods.
arXiv Detail & Related papers (2022-02-09T00:21:22Z) - A Survey on Causal Inference [64.45536158710014]
Causal inference is a critical research topic across many domains, such as statistics, computer science, education, public policy and economics.
Various causal effect estimation methods for observational data have sprung up.
arXiv Detail & Related papers (2020-02-05T21:35:29Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.