A Typology of Data Anomalies
- URL: http://arxiv.org/abs/2107.01615v1
- Date: Sun, 4 Jul 2021 13:12:24 GMT
- Title: A Typology of Data Anomalies
- Authors: Ralph Foorthuis
- Abstract summary: Anomalies are cases that are in some way unusual and do not appear to fit the general patterns present in the dataset.
This paper introduces a general typology of anomalies that offers a clear and tangible definition of the different types of anomalies in datasets.
- Score: 0.0
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Anomalies are cases that are in some way unusual and do not appear to fit the
general patterns present in the dataset. Several conceptualizations exist to
distinguish between different types of anomalies. However, these are either too
specific to be generally applicable or so abstract that they neither provide
concrete insight into the nature of anomaly types nor facilitate the functional
evaluation of anomaly detection algorithms. With the recent criticism on 'black
box' algorithms and analytics it has become clear that this is an undesirable
situation. This paper therefore introduces a general typology of anomalies that
offers a clear and tangible definition of the different types of anomalies in
datasets. The typology also facilitates the evaluation of the functional
capabilities of anomaly detection algorithms and as a framework assists in
analyzing the conceptual levels of data, patterns and anomalies. Finally, it
serves as an analytical tool for studying anomaly types from other typologies.
Related papers
- Reimagining Anomalies: What If Anomalies Were Normal? [21.480869966442143]
We introduce a novel explanation method that generates multiple counterfactual examples for each anomaly.
A counterfactual example is a modification of the anomaly that is perceived as normal by the anomaly detector.
The method provides a high-level semantic explanation of the mechanism that triggered the anomaly detector, allowing users to explore "what-if scenarios"
arXiv Detail & Related papers (2024-02-22T11:56:44Z) - Deep Learning for Time Series Anomaly Detection: A Survey [53.83593870825628]
Time series anomaly detection has applications in a wide range of research fields and applications, including manufacturing and healthcare.
The large size and complex patterns of time series have led researchers to develop specialised deep learning models for detecting anomalous patterns.
This survey focuses on providing structured and comprehensive state-of-the-art time series anomaly detection models through the use of deep learning.
arXiv Detail & Related papers (2022-11-09T22:40:22Z) - Catching Both Gray and Black Swans: Open-set Supervised Anomaly
Detection [90.32910087103744]
A few labeled anomaly examples are often available in many real-world applications.
These anomaly examples provide valuable knowledge about the application-specific abnormality.
Those anomalies seen during training often do not illustrate every possible class of anomaly.
This paper tackles open-set supervised anomaly detection.
arXiv Detail & Related papers (2022-03-28T05:21:37Z) - A Taxonomy of Anomalies in Log Data [0.09558392439655014]
A common taxonomy for anomalies already exists, but it has not yet been applied specifically to log data.
We present a taxonomy for different kinds of log data anomalies and introduce a method for analyzing such anomalies in labeled datasets.
Our results show, that the most common anomaly type is also the easiest to predict.
arXiv Detail & Related papers (2021-11-26T12:23:06Z) - Variation and generality in encoding of syntactic anomaly information in
sentence embeddings [7.132368785057315]
We explore fine-grained differences in anomaly encoding by designing probing tasks that vary the hierarchical level at which anomalies occur in a sentence.
We test not only models' ability to detect a given anomaly, but also the generality of the detected anomaly signal.
Results suggest that all models encode some information supporting anomaly detection, but detection performance varies between anomalies.
arXiv Detail & Related papers (2021-11-12T10:23:43Z) - Explainable Deep Few-shot Anomaly Detection with Deviation Networks [123.46611927225963]
We introduce a novel weakly-supervised anomaly detection framework to train detection models.
The proposed approach learns discriminative normality by leveraging the labeled anomalies and a prior probability.
Our model is substantially more sample-efficient and robust, and performs significantly better than state-of-the-art competing methods in both closed-set and open-set settings.
arXiv Detail & Related papers (2021-08-01T14:33:17Z) - Toward Deep Supervised Anomaly Detection: Reinforcement Learning from
Partially Labeled Anomaly Data [150.9270911031327]
We consider the problem of anomaly detection with a small set of partially labeled anomaly examples and a large-scale unlabeled dataset.
Existing related methods either exclusively fit the limited anomaly examples that typically do not span the entire set of anomalies, or proceed with unsupervised learning from the unlabeled data.
We propose here instead a deep reinforcement learning-based approach that enables an end-to-end optimization of the detection of both labeled and unlabeled anomalies.
arXiv Detail & Related papers (2020-09-15T03:05:39Z) - On the Nature and Types of Anomalies: A Review of Deviations in Data [0.0]
This study offers the first theoretically principled and domain-independent typology of data anomalies.
To concretely define the concept of the anomaly and its different manifestations, the typology employs five dimensions.
These fundamental and data-centric dimensions naturally yield 3 broad groups, 9 basic types, and 63 subtypes of anomalies.
arXiv Detail & Related papers (2020-07-30T17:55:11Z) - Manifolds for Unsupervised Visual Anomaly Detection [79.22051549519989]
Unsupervised learning methods that don't necessarily encounter anomalies in training would be immensely useful.
We develop a novel hyperspherical Variational Auto-Encoder (VAE) via stereographic projections with a gyroplane layer.
We present state-of-the-art results on visual anomaly benchmarks in precision manufacturing and inspection, demonstrating real-world utility in industrial AI scenarios.
arXiv Detail & Related papers (2020-06-19T20:41:58Z) - Deep Weakly-supervised Anomaly Detection [118.55172352231381]
Pairwise Relation prediction Network (PReNet) learns pairwise relation features and anomaly scores.
PReNet can detect any seen/unseen abnormalities that fit the learned pairwise abnormal patterns.
Empirical results on 12 real-world datasets show that PReNet significantly outperforms nine competing methods in detecting seen and unseen anomalies.
arXiv Detail & Related papers (2019-10-30T00:40:25Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.