How to find a unicorn: a novel model-free, unsupervised anomaly
detection method for time series
- URL: http://arxiv.org/abs/2004.11468v3
- Date: Tue, 15 Jun 2021 09:08:02 GMT
- Title: How to find a unicorn: a novel model-free, unsupervised anomaly
detection method for time series
- Authors: Zsigmond Benk\H{o}, Tam\'as B\'abel, Zolt\'an Somogyv\'ari
- Abstract summary: We introduce a new anomaly concept called "unicorn" or unique event and present a new, model-free, unsupervised detection algorithm to detect unicorns.
The performance of our algorithm was examined in recognizing unique events on different types of simulated data sets with anomalies.
- Score: 0.0
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Recognition of anomalous events is a challenging but critical task in many
scientific and industrial fields, especially when the properties of anomalies
are unknown. In this paper, we introduce a new anomaly concept called "unicorn"
or unique event and present a new, model-free, unsupervised detection algorithm
to detect unicorns. The key component of the new algorithm is the Temporal
Outlier Factor (TOF) to measure the uniqueness of events in continuous data
sets from dynamic systems. The concept of unique events differs significantly
from traditional outliers in many aspects: while repetitive outliers are no
longer unique events, a unique event is not necessarily an outlier; it does not
necessarily fall out from the distribution of normal activity. The performance
of our algorithm was examined in recognizing unique events on different types
of simulated data sets with anomalies and it was compared with the Local
Outlier Factor (LOF) and discord discovery algorithms. TOF had superior
performance compared to LOF and discord algorithms even in recognizing
traditional outliers and it also recognized unique events that those did not.
The benefits of the unicorn concept and the new detection method were
illustrated by example data sets from very different scientific fields. Our
algorithm successfully recognized unique events in those cases where they were
already known such as the gravitational waves of a binary black hole merger on
LIGO detector data and the signs of respiratory failure on ECG data series.
Furthermore, unique events were found on the LIBOR data set of the last 30
years.
Related papers
- Abnormal Event Detection via Hypergraph Contrastive Learning [54.80429341415227]
Abnormal event detection plays an important role in many real applications.
In this paper, we study the unsupervised abnormal event detection problem in Attributed Heterogeneous Information Network.
A novel hypergraph contrastive learning method, named AEHCL, is proposed to fully capture abnormal event patterns.
arXiv Detail & Related papers (2023-04-02T08:23:20Z) - Towards Dynamic Causal Discovery with Rare Events: A Nonparametric
Conditional Independence Test [4.67306371596399]
We introduce a novel statistical independence test on data collected from time-invariant systems in which rare but consequential events occur.
We provide non-asymptotic sample bounds for the consistency of our method, and validate its performance across various simulated and real-world datasets.
arXiv Detail & Related papers (2022-11-29T21:15:51Z) - Are we certain it's anomalous? [57.729669157989235]
Anomaly detection in time series is a complex task since anomalies are rare due to highly non-linear temporal correlations.
Here we propose the novel use of Hyperbolic uncertainty for Anomaly Detection (HypAD)
HypAD learns self-supervisedly to reconstruct the input signal.
arXiv Detail & Related papers (2022-11-16T21:31:39Z) - Anomaly Rule Detection in Sequence Data [2.3757190901941736]
We present a new anomaly detection framework called DUOS that enables Discovery of Utility-aware Outlier Sequential rules from a set of sequences.
In this work, we incorporate both the anomalousness and utility of a group, and then introduce the concept of utility-aware outlier rule (UOSR)
arXiv Detail & Related papers (2021-11-29T23:52:31Z) - New Methods and Datasets for Group Anomaly Detection From Fundamental
Physics [0.4297070083645048]
Unsupervised group anomaly detection has become a new frontier of fundamental physics.
We propose a realistic synthetic benchmark dataset (LHCO 2020) for the development of group anomaly detection algorithms.
arXiv Detail & Related papers (2021-07-06T18:00:57Z) - Recomposition vs. Prediction: A Novel Anomaly Detection for Discrete
Events Based On Autoencoder [5.781280693720236]
One of the most challenging problems in the field of intrusion detection is anomaly detection for discrete event logs.
We propose DabLog, a Deep Autoencoder-Based anomaly detection method for discrete event Logs.
Our approach determines whether a sequence is normal or abnormal by analyzing (encoding) and reconstructing (decoding) the given sequence.
arXiv Detail & Related papers (2020-12-27T16:31:05Z) - TadGAN: Time Series Anomaly Detection Using Generative Adversarial
Networks [73.01104041298031]
TadGAN is an unsupervised anomaly detection approach built on Generative Adversarial Networks (GANs)
To capture the temporal correlations of time series, we use LSTM Recurrent Neural Networks as base models for Generators and Critics.
To demonstrate the performance and generalizability of our approach, we test several anomaly scoring techniques and report the best-suited one.
arXiv Detail & Related papers (2020-09-16T15:52:04Z) - Toward Deep Supervised Anomaly Detection: Reinforcement Learning from
Partially Labeled Anomaly Data [150.9270911031327]
We consider the problem of anomaly detection with a small set of partially labeled anomaly examples and a large-scale unlabeled dataset.
Existing related methods either exclusively fit the limited anomaly examples that typically do not span the entire set of anomalies, or proceed with unsupervised learning from the unlabeled data.
We propose here instead a deep reinforcement learning-based approach that enables an end-to-end optimization of the detection of both labeled and unlabeled anomalies.
arXiv Detail & Related papers (2020-09-15T03:05:39Z) - Multi-Scale One-Class Recurrent Neural Networks for Discrete Event
Sequence Anomaly Detection [63.825781848587376]
We propose OC4Seq, a one-class recurrent neural network for detecting anomalies in discrete event sequences.
Specifically, OC4Seq embeds the discrete event sequences into latent spaces, where anomalies can be easily detected.
arXiv Detail & Related papers (2020-08-31T04:48:22Z) - Deep Weakly-supervised Anomaly Detection [118.55172352231381]
Pairwise Relation prediction Network (PReNet) learns pairwise relation features and anomaly scores.
PReNet can detect any seen/unseen abnormalities that fit the learned pairwise abnormal patterns.
Empirical results on 12 real-world datasets show that PReNet significantly outperforms nine competing methods in detecting seen and unseen anomalies.
arXiv Detail & Related papers (2019-10-30T00:40:25Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.