Testing Causality in Scientific Modelling Software
- URL: http://arxiv.org/abs/2209.00357v2
- Date: Fri, 30 Jun 2023 14:01:25 GMT
- Title: Testing Causality in Scientific Modelling Software
- Authors: Andrew G. Clark, Michael Foster, Benedikt Prifling, Neil Walkinshaw,
Robert M. Hierons, Volker Schmidt, Robert D. Turner
- Abstract summary: Causal Testing Framework is a framework that uses Causal Inference techniques to establish causal effects from existing data.
We present three case studies covering real-world scientific models, demonstrating how the Causal Testing Framework can infer metamorphic test outcomes.
- Score: 0.26388783516590225
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: From simulating galaxy formation to viral transmission in a pandemic,
scientific models play a pivotal role in developing scientific theories and
supporting government policy decisions that affect us all. Given these critical
applications, a poor modelling assumption or bug could have far-reaching
consequences. However, scientific models possess several properties that make
them notoriously difficult to test, including a complex input space, long
execution times, and non-determinism, rendering existing testing techniques
impractical. In fields such as epidemiology, where researchers seek answers to
challenging causal questions, a statistical methodology known as Causal
Inference has addressed similar problems, enabling the inference of causal
conclusions from noisy, biased, and sparse data instead of costly experiments.
This paper introduces the Causal Testing Framework: a framework that uses
Causal Inference techniques to establish causal effects from existing data,
enabling users to conduct software testing activities concerning the effect of
a change, such as Metamorphic Testing, a posteriori. We present three case
studies covering real-world scientific models, demonstrating how the Causal
Testing Framework can infer metamorphic test outcomes from reused, confounded
test data to provide an efficient solution for testing scientific modelling
software.
Related papers
- DAG-aware Transformer for Causal Effect Estimation [0.8192907805418583]
Causal inference is a critical task across fields such as healthcare, economics, and the social sciences.
In this paper, we present a novel transformer-based method for causal inference that overcomes these challenges.
The core innovation of our model lies in its integration of causal Directed Acyclic Graphs (DAGs) directly into the attention mechanism.
arXiv Detail & Related papers (2024-10-13T23:17:58Z) - Causal Representation Learning in Temporal Data via Single-Parent Decoding [66.34294989334728]
Scientific research often seeks to understand the causal structure underlying high-level variables in a system.
Scientists typically collect low-level measurements, such as geographically distributed temperature readings.
We propose a differentiable method, Causal Discovery with Single-parent Decoding, that simultaneously learns the underlying latents and a causal graph over them.
arXiv Detail & Related papers (2024-10-09T15:57:50Z) - Hypothesizing Missing Causal Variables with LLMs [55.28678224020973]
We formulate a novel task where the input is a partial causal graph with missing variables, and the output is a hypothesis about the missing variables to complete the partial graph.
We show the strong ability of LLMs to hypothesize the mediation variables between a cause and its effect.
We also observe surprising results where some of the open-source models outperform the closed GPT-4 model.
arXiv Detail & Related papers (2024-09-04T10:37:44Z) - Smoke and Mirrors in Causal Downstream Tasks [59.90654397037007]
This paper looks at the causal inference task of treatment effect estimation, where the outcome of interest is recorded in high-dimensional observations.
We compare 6 480 models fine-tuned from state-of-the-art visual backbones, and find that the sampling and modeling choices significantly affect the accuracy of the causal estimate.
Our results suggest that future benchmarks should carefully consider real downstream scientific questions, especially causal ones.
arXiv Detail & Related papers (2024-05-27T13:26:34Z) - The Causal Chambers: Real Physical Systems as a Testbed for AI Methodology [10.81691411087626]
In some fields of AI, machine learning and statistics, the validation of new methods and algorithms is often hindered by the scarcity of suitable real-world datasets.
We have constructed two devices that allow us to quickly and inexpensively produce large datasets from non-trivial but well-understood physical systems.
arXiv Detail & Related papers (2024-04-17T13:00:52Z) - Targeted Reduction of Causal Models [55.11778726095353]
Causal Representation Learning offers a promising avenue to uncover interpretable causal patterns in simulations.
We introduce Targeted Causal Reduction (TCR), a method for condensing complex intervenable models into a concise set of causal factors.
Its ability to generate interpretable high-level explanations from complex models is demonstrated on toy and mechanical systems.
arXiv Detail & Related papers (2023-11-30T15:46:22Z) - Partially Specified Causal Simulations [0.0]
Many causal inference literature tend to design over-restricted or misspecified studies.
We introduce partially randomized causal simulation (PARCS), a simulation framework that meets those desiderata.
We reproduce and extend the simulation studies of two well-known causal discovery and missing data analysis papers.
arXiv Detail & Related papers (2023-09-19T10:50:35Z) - GFlowNets for AI-Driven Scientific Discovery [74.27219800878304]
We present a new probabilistic machine learning framework called GFlowNets.
GFlowNets can be applied in the modeling, hypotheses generation and experimental design stages of the experimental science loop.
We argue that GFlowNets can become a valuable tool for AI-driven scientific discovery.
arXiv Detail & Related papers (2023-02-01T17:29:43Z) - Evaluating Causal Inference Methods [0.4588028371034407]
We introduce a deep generative model-based framework, Credence, to validate causal inference methods.
Our work introduces a deep generative model-based framework, Credence, to validate causal inference methods.
arXiv Detail & Related papers (2022-02-09T00:21:22Z) - A Survey on Causal Inference [64.45536158710014]
Causal inference is a critical research topic across many domains, such as statistics, computer science, education, public policy and economics.
Various causal effect estimation methods for observational data have sprung up.
arXiv Detail & Related papers (2020-02-05T21:35:29Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.