Metrics on Markov Equivalence Classes for Evaluating Causal Discovery Algorithms
- URL: http://arxiv.org/abs/2402.04952v2
- Date: Fri, 15 Mar 2024 17:21:56 GMT
- Title: Metrics on Markov Equivalence Classes for Evaluating Causal Discovery Algorithms
- Authors: Jonas Wahl, Jakob Runge,
- Abstract summary: We argue that an evaluation of a causal discovery method against synthetic data should include an analysis of how well this explicit goal is achieved.
We show that established evaluation measures do not accurately capture the difference in separations/connections of two causal graphs.
We introduce three new measures of distance called s/c-distance, Markov distance and Faithfulness distance that address this shortcoming.
- Score: 15.37737222790121
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Many state-of-the-art causal discovery methods aim to generate an output graph that encodes the graphical separation and connection statements of the causal graph that underlies the data-generating process. In this work, we argue that an evaluation of a causal discovery method against synthetic data should include an analysis of how well this explicit goal is achieved by measuring how closely the separations/connections of the method's output align with those of the ground truth. We show that established evaluation measures do not accurately capture the difference in separations/connections of two causal graphs, and we introduce three new measures of distance called s/c-distance, Markov distance and Faithfulness distance that address this shortcoming. We complement our theoretical analysis with toy examples, empirical experiments and pseudocode.
Related papers
- Adaptive Online Experimental Design for Causal Discovery [9.447864414136905]
Causal discovery aims to uncover cause-and-effect relationships encoded in causal graphs.
We focus on data interventional efficiency and formalize causal discovery from the perspective of online learning.
We propose a track-and-stop causal discovery algorithm that adaptively selects interventions from the graph separating system.
arXiv Detail & Related papers (2024-05-19T13:26:33Z) - A Generalization Theory of Cross-Modality Distillation with Contrastive Learning [49.35244441141323]
Cross-modality distillation arises as an important topic for data modalities containing limited knowledge.
We formulate a general framework of cross-modality contrastive distillation (CMCD), built upon contrastive learning.
Our algorithm outperforms existing algorithms consistently by a margin of 2-3% across diverse modalities and tasks.
arXiv Detail & Related papers (2024-05-06T11:05:13Z) - Collaborative Heterogeneous Causal Inference Beyond Meta-analysis [68.4474531911361]
We propose a collaborative inverse propensity score estimator for causal inference with heterogeneous data.
Our method shows significant improvements over the methods based on meta-analysis when heterogeneity increases.
arXiv Detail & Related papers (2024-04-24T09:04:36Z) - A continuous Structural Intervention Distance to compare Causal Graphs [5.477914707166288]
The distance is based on embedding intervention distributions over each pair of nodes.
We show theoretical results which we validate with numerical experiments on synthetic data.
arXiv Detail & Related papers (2023-07-31T07:20:26Z) - MAUVE Scores for Generative Models: Theory and Practice [95.86006777961182]
We present MAUVE, a family of comparison measures between pairs of distributions such as those encountered in the generative modeling of text or images.
We find that MAUVE can quantify the gaps between the distributions of human-written text and those of modern neural language models.
We demonstrate in the vision domain that MAUVE can identify known properties of generated images on par with or better than existing metrics.
arXiv Detail & Related papers (2022-12-30T07:37:40Z) - A Probabilistic Graph Coupling View of Dimension Reduction [5.35952718937799]
We introduce a unifying statistical framework based on the coupling of hidden graphs using cross entropy.
We show that existing pairwise similarity DR methods can be retrieved from our framework with particular choices of priors for the graphs.
Our model is leveraged and extended to address the issue while new links are drawn with Laplacian eigenmaps and PCA.
arXiv Detail & Related papers (2022-01-31T08:21:55Z) - Scalable Intervention Target Estimation in Linear Models [52.60799340056917]
Current approaches to causal structure learning either work with known intervention targets or use hypothesis testing to discover the unknown intervention targets.
This paper proposes a scalable and efficient algorithm that consistently identifies all intervention targets.
The proposed algorithm can be used to also update a given observational Markov equivalence class into the interventional Markov equivalence class.
arXiv Detail & Related papers (2021-11-15T03:16:56Z) - Graph Embedding with Data Uncertainty [113.39838145450007]
spectral-based subspace learning is a common data preprocessing step in many machine learning pipelines.
Most subspace learning methods do not take into consideration possible measurement inaccuracies or artifacts that can lead to data with high uncertainty.
arXiv Detail & Related papers (2020-09-01T15:08:23Z) - Towards Certified Robustness of Distance Metric Learning [53.96113074344632]
We advocate imposing an adversarial margin in the input space so as to improve the generalization and robustness of metric learning algorithms.
We show that the enlarged margin is beneficial to the generalization ability by using the theoretical technique of algorithmic robustness.
arXiv Detail & Related papers (2020-06-10T16:51:53Z) - A Ladder of Causal Distances [44.34185575573054]
We introduce a hierarchy of three distances, one for each rung of the "ladder of causation"
We put our causal distances to use by benchmarking standard causal discovery systems on both synthetic and real-world datasets.
Finally, we highlight the usefulness of our causal distances by briefly discussing further applications beyond the evaluation of causal discovery techniques.
arXiv Detail & Related papers (2020-05-05T20:39:07Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.