Towards Dynamic Causal Discovery with Rare Events: A Nonparametric Conditional Independence Test

• URL: http://arxiv.org/abs/2211.16596v5
• Date: Mon, 17 Jul 2023 22:24:16 GMT
• Title: Towards Dynamic Causal Discovery with Rare Events: A Nonparametric Conditional Independence Test
• Authors: Chih-Yuan Chiu, Kshitij Kulkarni, Shankar Sastry
• Abstract summary: We introduce a novel statistical independence test on data collected from time-invariant systems in which rare but consequential events occur. We provide non-asymptotic sample bounds for the consistency of our method, and validate its performance across various simulated and real-world datasets.
• Score: 4.67306371596399
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Causal phenomena associated with rare events occur across a wide range of engineering problems, such as risk-sensitive safety analysis, accident analysis and prevention, and extreme value theory. However, current methods for causal discovery are often unable to uncover causal links, between random variables in a dynamic setting, that manifest only when the variables first experience low-probability realizations. To address this issue, we introduce a novel statistical independence test on data collected from time-invariant dynamical systems in which rare but consequential events occur. In particular, we exploit the time-invariance of the underlying data to construct a superimposed dataset of the system state before rare events happen at different timesteps. We then design a conditional independence test on the reorganized data. We provide non-asymptotic sample complexity bounds for the consistency of our method, and validate its performance across various simulated and real-world datasets, including incident data collected from the Caltrans Performance Measurement System (PeMS). Code containing the datasets and experiments is publicly available.

Related papers

• DACR: Distribution-Augmented Contrastive Reconstruction for Time-Series Anomaly Detection [12.3866167448478]
  Anomaly detection in time-series data is crucial for identifying faults, failures, threats, and outliers across a range of applications. Recently, deep learning techniques have been applied to this topic, but they often struggle in real-world scenarios. We propose Distribution-Augmented Contrastive Reconstruction (DACR) to tackle these challenges.
  arXiv  Detail & Related papers  (2024-01-20T16:56:52Z)
• Graph Spatiotemporal Process for Multivariate Time Series Anomaly Detection with Missing Values [67.76168547245237]
  We introduce a novel framework called GST-Pro, which utilizes a graphtemporal process and anomaly scorer to detect anomalies. Our experimental results show that the GST-Pro method can effectively detect anomalies in time series data and outperforms state-of-the-art methods.
  arXiv  Detail & Related papers  (2024-01-11T10:10:16Z)
• Abnormal Event Detection via Hypergraph Contrastive Learning [54.80429341415227]
  Abnormal event detection plays an important role in many real applications. In this paper, we study the unsupervised abnormal event detection problem in Attributed Heterogeneous Information Network. A novel hypergraph contrastive learning method, named AEHCL, is proposed to fully capture abnormal event patterns.
  arXiv  Detail & Related papers  (2023-04-02T08:23:20Z)
• Causality-Based Multivariate Time Series Anomaly Detection [63.799474860969156]
  We formulate the anomaly detection problem from a causal perspective and view anomalies as instances that do not follow the regular causal mechanism to generate the multivariate data. We then propose a causality-based anomaly detection approach, which first learns the causal structure from data and then infers whether an instance is an anomaly relative to the local causal mechanism. We evaluate our approach with both simulated and public datasets as well as a case study on real-world AIOps applications.
  arXiv  Detail & Related papers  (2022-06-30T06:00:13Z)
• Detecting hidden confounding in observational data using multiple environments [0.81585306387285]
  We present a theory for testable conditional independencies that are only absent when there is hidden confounding. In most cases, the proposed procedure correctly predicts the presence of hidden confounding.
  arXiv  Detail & Related papers  (2022-05-27T12:20:09Z)
• Equivariance Allows Handling Multiple Nuisance Variables When Analyzing Pooled Neuroimaging Datasets [53.34152466646884]
  In this paper, we show how bringing recent results on equivariant representation learning instantiated on structured spaces together with simple use of classical results on causal inference provides an effective practical solution. We demonstrate how our model allows dealing with more than one nuisance variable under some assumptions and can enable analysis of pooled scientific datasets in scenarios that would otherwise entail removing a large portion of the samples.
  arXiv  Detail & Related papers  (2022-03-29T04:54:06Z)
• Federated Variational Learning for Anomaly Detection in Multivariate Time Series [13.328883578980237]
  We propose an unsupervised time series anomaly detection framework in a federated fashion. We leave the training data distributed at the edge to learn a shared Variational Autoencoder (VAE) based on Convolutional Gated Recurrent Unit (ConvGRU) model. Experiments on three real-world networked sensor datasets illustrate the advantage of our approach over other state-of-the-art models.
  arXiv  Detail & Related papers  (2021-08-18T22:23:15Z)
• On the Sample Complexity of Causal Discovery and the Value of Domain Expertise [0.0]
  Causal discovery methods seek to identify causal relations between random variables from purely observational data. In this paper, we analyze the sample complexity of causal discovery algorithms without a CI oracle. Our methods allow us to quantify the value of domain expertise in terms of data samples.
  arXiv  Detail & Related papers  (2021-02-05T16:26:17Z)
• TadGAN: Time Series Anomaly Detection Using Generative Adversarial Networks [73.01104041298031]
  TadGAN is an unsupervised anomaly detection approach built on Generative Adversarial Networks (GANs) To capture the temporal correlations of time series, we use LSTM Recurrent Neural Networks as base models for Generators and Critics. To demonstrate the performance and generalizability of our approach, we test several anomaly scoring techniques and report the best-suited one.
  arXiv  Detail & Related papers  (2020-09-16T15:52:04Z)
• Multi-Scale One-Class Recurrent Neural Networks for Discrete Event Sequence Anomaly Detection [63.825781848587376]
  We propose OC4Seq, a one-class recurrent neural network for detecting anomalies in discrete event sequences. Specifically, OC4Seq embeds the discrete event sequences into latent spaces, where anomalies can be easily detected.
  arXiv  Detail & Related papers  (2020-08-31T04:48:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.