Assumption violations in causal discovery and the robustness of score matching

• URL: http://arxiv.org/abs/2310.13387v2
• Date: Thu, 26 Sep 2024 17:55:48 GMT
• Title: Assumption violations in causal discovery and the robustness of score matching
• Authors: Francesco Montagna, Atalanti A. Mastakouri, Elias Eulig, Nicoletta Noceti, Lorenzo Rosasco, Dominik Janzing, Bryon Aragam, Francesco Locatello,
• Abstract summary: This paper extensively benchmarks the empirical performance of recent causal discovery methods on observational i.i.d. data. We show that score matching-based methods demonstrate surprising performance in the false positive and false negative rate of the inferred graph. We hope this paper will set a new standard for the evaluation of causal discovery methods.
• Score: 38.60630271550033
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: When domain knowledge is limited and experimentation is restricted by ethical, financial, or time constraints, practitioners turn to observational causal discovery methods to recover the causal structure, exploiting the statistical properties of their data. Because causal discovery without further assumptions is an ill-posed problem, each algorithm comes with its own set of usually untestable assumptions, some of which are hard to meet in real datasets. Motivated by these considerations, this paper extensively benchmarks the empirical performance of recent causal discovery methods on observational i.i.d. data generated under different background conditions, allowing for violations of the critical assumptions required by each selected approach. Our experimental findings show that score matching-based methods demonstrate surprising performance in the false positive and false negative rate of the inferred graph in these challenging scenarios, and we provide theoretical insights into their performance. This work is also the first effort to benchmark the stability of causal discovery algorithms with respect to the values of their hyperparameters. Finally, we hope this paper will set a new standard for the evaluation of causal discovery methods and can serve as an accessible entry point for practitioners interested in the field, highlighting the empirical implications of different algorithm choices.

Related papers

• Unsupervised Pairwise Causal Discovery on Heterogeneous Data using Mutual Information Measures [49.1574468325115]
  Causal Discovery is a technique that tackles the challenge by analyzing the statistical properties of the constituent variables. We question the current (possibly misleading) baseline results on the basis that they were obtained through supervised learning. In consequence, we approach this problem in an unsupervised way, using robust Mutual Information measures.
  arXiv  Detail & Related papers  (2024-08-01T09:11:08Z)
• Benchmarking Bayesian Causal Discovery Methods for Downstream Treatment Effect Estimation [137.3520153445413]
  A notable gap exists in the evaluation of causal discovery methods, where insufficient emphasis is placed on downstream inference. We evaluate seven established baseline causal discovery methods including a newly proposed method based on GFlowNets. The results of our study demonstrate that some of the algorithms studied are able to effectively capture a wide range of useful and diverse ATE modes.
  arXiv  Detail & Related papers  (2023-07-11T02:58:10Z)
• Systematic Evaluation of Predictive Fairness [60.0947291284978]
  Mitigating bias in training on biased datasets is an important open problem. We examine the performance of various debiasing methods across multiple tasks. We find that data conditions have a strong influence on relative model performance.
  arXiv  Detail & Related papers  (2022-10-17T05:40:13Z)
• Valid Inference After Causal Discovery [73.87055989355737]
  We develop tools for valid post-causal-discovery inference. We show that a naive combination of causal discovery and subsequent inference algorithms leads to highly inflated miscoverage rates.
  arXiv  Detail & Related papers  (2022-08-11T17:40:45Z)
• Empirical Bayesian Approaches for Robust Constraint-based Causal Discovery under Insufficient Data [38.883810061897094]
  Causal discovery methods assume data sufficiency, which may not be the case in many real world datasets. We propose Bayesian-augmented frequentist independence tests to improve the performance of constraint-based causal discovery methods under insufficient data. Experiments show significant performance improvement in terms of both accuracy and efficiency over SOTA methods.
  arXiv  Detail & Related papers  (2022-06-16T21:08:49Z)
• Generalization bounds and algorithms for estimating conditional average treatment effect of dosage [13.867315751451494]
  We investigate the task of estimating the conditional average causal effect of treatment-dosage pairs from a combination of observational data and assumptions on the causal relationships in the underlying system. This has been a longstanding challenge for fields of study such as epidemiology or economics that require a treatment-dosage pair to make decisions. We show empirically new state-of-the-art performance results across several benchmark datasets for this problem.
  arXiv  Detail & Related papers  (2022-05-29T15:26:59Z)
• Evaluating Causal Inference Methods [0.4588028371034407]
  We introduce a deep generative model-based framework, Credence, to validate causal inference methods. Our work introduces a deep generative model-based framework, Credence, to validate causal inference methods.
  arXiv  Detail & Related papers  (2022-02-09T00:21:22Z)
• Doing Great at Estimating CATE? On the Neglected Assumptions in Benchmark Comparisons of Treatment Effect Estimators [91.3755431537592]
  We show that even in arguably the simplest setting, estimation under ignorability assumptions can be misleading. We consider two popular machine learning benchmark datasets for evaluation of heterogeneous treatment effect estimators. We highlight that the inherent characteristics of the benchmark datasets favor some algorithms over others.
  arXiv  Detail & Related papers  (2021-07-28T13:21:27Z)
• Data Generating Process to Evaluate Causal Discovery Techniques for Time Series Data [1.5293427903448025]
  We propose a framework for developing, evaluating, and benchmarking time series causal discovery methods. The framework can be used to fine tune novel methods on vast amounts of data, without "overfitting" them to a benchmark. Using our framework, we evaluate prominent time series causal discovery methods and demonstrate a notable degradation in performance when their assumptions are invalidated.
  arXiv  Detail & Related papers  (2021-04-16T11:38:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.