Real-time Anomaly Detection for Multivariate Data Streams
- URL: http://arxiv.org/abs/2209.12398v1
- Date: Mon, 26 Sep 2022 03:40:37 GMT
- Title: Real-time Anomaly Detection for Multivariate Data Streams
- Authors: Kenneth Odoh
- Abstract summary: We present a real-time multivariate anomaly detection algorithm for data streams based on the Probabilistic Exponentially Weighted Moving Average (PEWMA)
Our formulation is resilient to (abrupt transient, abrupt distributional, and gradual distributional) shifts in the data.
Our proposed anomaly detection algorithm works in an unsupervised manner eliminating the need for labeled examples.
- Score: 0.0
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: We present a real-time multivariate anomaly detection algorithm for data
streams based on the Probabilistic Exponentially Weighted Moving Average
(PEWMA). Our formulation is resilient to (abrupt transient, abrupt
distributional, and gradual distributional) shifts in the data. The novel
anomaly detection routines utilize an incremental online algorithm to handle
streams. Furthermore, our proposed anomaly detection algorithm works in an
unsupervised manner eliminating the need for labeled examples. Our algorithm
performs well and is resilient in the face of concept drifts.
Related papers
- A Mirror Descent-Based Algorithm for Corruption-Tolerant Distributed Gradient Descent [57.64826450787237]
We show how to analyze the behavior of distributed gradient descent algorithms in the presence of adversarial corruptions.
We show how to use ideas from (lazy) mirror descent to design a corruption-tolerant distributed optimization algorithm.
Experiments based on linear regression, support vector classification, and softmax classification on the MNIST dataset corroborate our theoretical findings.
arXiv Detail & Related papers (2024-07-19T08:29:12Z) - Partially-Observable Sequential Change-Point Detection for Autocorrelated Data via Upper Confidence Region [12.645304808491309]
We propose a detection scheme called adaptive upper confidence region with state space model (AUCRSS) for sequential change point detection.
A partially-observable Kalman filter algorithm is developed for online inference of SSM, and accordingly, a change point detection scheme based on a generalized likelihood ratio test is analyzed.
arXiv Detail & Related papers (2024-03-30T02:32:53Z) - Graph Spatiotemporal Process for Multivariate Time Series Anomaly
Detection with Missing Values [67.76168547245237]
We introduce a novel framework called GST-Pro, which utilizes a graphtemporal process and anomaly scorer to detect anomalies.
Our experimental results show that the GST-Pro method can effectively detect anomalies in time series data and outperforms state-of-the-art methods.
arXiv Detail & Related papers (2024-01-11T10:10:16Z) - Low-rank extended Kalman filtering for online learning of neural
networks from streaming data [71.97861600347959]
We propose an efficient online approximate Bayesian inference algorithm for estimating the parameters of a nonlinear function from a potentially non-stationary data stream.
The method is based on the extended Kalman filter (EKF), but uses a novel low-rank plus diagonal decomposition of the posterior matrix.
In contrast to methods based on variational inference, our method is fully deterministic, and does not require step-size tuning.
arXiv Detail & Related papers (2023-05-31T03:48:49Z) - Optimal Algorithms for the Inhomogeneous Spiked Wigner Model [89.1371983413931]
We derive an approximate message-passing algorithm (AMP) for the inhomogeneous problem.
We identify in particular the existence of a statistical-to-computational gap where known algorithms require a signal-to-noise ratio bigger than the information-theoretic threshold to perform better than random.
arXiv Detail & Related papers (2023-02-13T19:57:17Z) - Real-Time Outlier Detection with Dynamic Process Limits [0.609170287691728]
This paper proposes an online anomaly detection algorithm for existing real-time infrastructures.
Online inverse cumulative distribution-based approach is introduced to eliminate common problems of offline anomaly detectors.
The benefit of the proposed method is the ease of use, fast computation, and deployability as shown in two case studies of real microgrid operation data.
arXiv Detail & Related papers (2023-01-31T10:23:02Z) - InQMAD: Incremental Quantum Measurement Anomaly Detection [0.0]
Streaming anomaly detection refers to the problem of detecting anomalous data samples in streams of data.
This problem poses challenges that classical and deep anomaly detection methods are not designed to cope with.
We present a new incremental anomaly detection method that performs continuous density estimation.
arXiv Detail & Related papers (2022-10-11T00:22:34Z) - A Robust and Explainable Data-Driven Anomaly Detection Approach For
Power Electronics [56.86150790999639]
We present two anomaly detection and classification approaches, namely the Matrix Profile algorithm and anomaly transformer.
The Matrix Profile algorithm is shown to be well suited as a generalizable approach for detecting real-time anomalies in streaming time-series data.
A series of custom filters is created and added to the detector to tune its sensitivity, recall, and detection accuracy.
arXiv Detail & Related papers (2022-09-23T06:09:35Z) - An Efficient Anomaly Detection Approach using Cube Sampling with
Streaming Data [2.0515785954568626]
Anomaly detection is critical in various fields, including intrusion detection, health monitoring, fault diagnosis, and sensor network event detection.
The isolation forest (or iForest) approach is a well-known technique for detecting anomalies.
We propose an efficient iForest based approach for anomaly detection using cube sampling that is effective on streaming data.
arXiv Detail & Related papers (2021-10-05T04:23:00Z) - Machine Learning for Online Algorithm Selection under Censored Feedback [71.6879432974126]
In online algorithm selection (OAS), instances of an algorithmic problem class are presented to an agent one after another, and the agent has to quickly select a presumably best algorithm from a fixed set of candidate algorithms.
For decision problems such as satisfiability (SAT), quality typically refers to the algorithm's runtime.
In this work, we revisit multi-armed bandit algorithms for OAS and discuss their capability of dealing with the problem.
We adapt them towards runtime-oriented losses, allowing for partially censored data while keeping a space- and time-complexity independent of the time horizon.
arXiv Detail & Related papers (2021-09-13T18:10:52Z) - Anomaly Detection in Trajectory Data with Normalizing Flows [0.0]
We propose an approach based on normalizing flows that enables complex density estimation from data with neural networks.
Our proposal computes exact model likelihood values, an important feature of normalizing flows, for each segment of the trajectory.
We evaluate our methodology, named aggregated anomaly detection with normalizing flows (GRADINGS), using real world trajectory data and compare it with more traditional anomaly detection techniques.
arXiv Detail & Related papers (2020-04-13T14:16:40Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.