Sub-clusters of Normal Data for Anomaly Detection
- URL: http://arxiv.org/abs/2011.08408v1
- Date: Tue, 17 Nov 2020 03:53:31 GMT
- Title: Sub-clusters of Normal Data for Anomaly Detection
- Authors: Gahye Lee and Seungkyu Lee
- Abstract summary: Anomaly detection in data analysis is an interesting but still challenging research topic in real world applications.
Existing anomaly detection methods show limited performances with high dimensional data such as ImageNet.
In this paper, we study anomaly detection with high dimensional and complex normal data.
- Score: 0.15229257192293197
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Anomaly detection in data analysis is an interesting but still challenging
research topic in real world applications. As the complexity of data dimension
increases, it requires to understand the semantic contexts in its description
for effective anomaly characterization. However, existing anomaly detection
methods show limited performances with high dimensional data such as ImageNet.
Existing studies have evaluated their performance on low dimensional, clean and
well separated data set such as MNIST and CIFAR-10. In this paper, we study
anomaly detection with high dimensional and complex normal data. Our
observation is that, in general, anomaly data is defined by semantically
explainable features which are able to be used in defining semantic
sub-clusters of normal data as well. We hypothesize that if there exists
reasonably good feature space semantically separating sub-clusters of given
normal data, unseen anomaly also can be well distinguished in the space from
the normal data. We propose to perform semantic clustering on given normal data
and train a classifier to learn the discriminative feature space where anomaly
detection is finally performed. Based on our careful and extensive experimental
evaluations with MNIST, CIFAR-10, and ImageNet with various combinations of
normal and anomaly data, we show that our anomaly detection scheme outperforms
state of the art methods especially with high dimensional real world images.
Related papers
- Anomaly Detection by Context Contrasting [57.695202846009714]
Anomaly Detection focuses on identifying samples that deviate from the norm.
Recent advances in self-supervised learning have shown great promise in this regard.
We propose Con2, which addresses this problem by setting normal training data into distinct contexts.
Our approach achieves state-of-the-art performance on various benchmarks while exhibiting superior performance in a more realistic healthcare setting.
arXiv Detail & Related papers (2024-05-29T07:59:06Z) - ARC: A Generalist Graph Anomaly Detector with In-Context Learning [62.202323209244]
ARC is a generalist GAD approach that enables a one-for-all'' GAD model to detect anomalies across various graph datasets on-the-fly.
equipped with in-context learning, ARC can directly extract dataset-specific patterns from the target dataset.
Extensive experiments on multiple benchmark datasets from various domains demonstrate the superior anomaly detection performance, efficiency, and generalizability of ARC.
arXiv Detail & Related papers (2024-05-27T02:42:33Z) - AGAD: Adversarial Generative Anomaly Detection [12.68966318231776]
Anomaly detection suffered from the lack of anomalies due to the diversity of abnormalities and the difficulties of obtaining large-scale anomaly data.
We propose Adversarial Generative Anomaly Detection (AGAD), a self-contrast-based anomaly detection paradigm.
Our method generates pseudo-anomaly data for both supervised and semi-supervised anomaly detection scenarios.
arXiv Detail & Related papers (2023-04-09T10:40:02Z) - Perturbation Learning Based Anomaly Detection [19.41730292017383]
The main idea is to learn small perturbations to perturb normal data and learn a classifier to classify the normal data and the perturbed data into two different classes.
The perturbations should be as small as possible but the classifier is still able to recognize the perturbed data from unperturbed data.
Compared with the state-of-the-art of anomaly detection, our method does not require any assumption about the shape of the decision boundary.
arXiv Detail & Related papers (2022-06-06T16:01:01Z) - Catching Both Gray and Black Swans: Open-set Supervised Anomaly
Detection [90.32910087103744]
A few labeled anomaly examples are often available in many real-world applications.
These anomaly examples provide valuable knowledge about the application-specific abnormality.
Those anomalies seen during training often do not illustrate every possible class of anomaly.
This paper tackles open-set supervised anomaly detection.
arXiv Detail & Related papers (2022-03-28T05:21:37Z) - Explainable Deep Few-shot Anomaly Detection with Deviation Networks [123.46611927225963]
We introduce a novel weakly-supervised anomaly detection framework to train detection models.
The proposed approach learns discriminative normality by leveraging the labeled anomalies and a prior probability.
Our model is substantially more sample-efficient and robust, and performs significantly better than state-of-the-art competing methods in both closed-set and open-set settings.
arXiv Detail & Related papers (2021-08-01T14:33:17Z) - DASVDD: Deep Autoencoding Support Vector Data Descriptor for Anomaly
Detection [9.19194451963411]
Semi-supervised anomaly detection aims to detect anomalies from normal samples using a model that is trained on normal data.
We propose a method, DASVDD, that jointly learns the parameters of an autoencoder while minimizing the volume of an enclosing hyper-sphere on its latent representation.
arXiv Detail & Related papers (2021-06-09T21:57:41Z) - Toward Deep Supervised Anomaly Detection: Reinforcement Learning from
Partially Labeled Anomaly Data [150.9270911031327]
We consider the problem of anomaly detection with a small set of partially labeled anomaly examples and a large-scale unlabeled dataset.
Existing related methods either exclusively fit the limited anomaly examples that typically do not span the entire set of anomalies, or proceed with unsupervised learning from the unlabeled data.
We propose here instead a deep reinforcement learning-based approach that enables an end-to-end optimization of the detection of both labeled and unlabeled anomalies.
arXiv Detail & Related papers (2020-09-15T03:05:39Z) - Factor Analysis of Mixed Data for Anomaly Detection [5.77019633619109]
Anomalous observations may correspond to financial fraud, health risks, or incorrectly measured data in practice.
We show detecting anomalies in high-dimensional mixed data is enhanced through first embedding the data then assessing an anomaly scoring scheme.
arXiv Detail & Related papers (2020-05-25T14:13:10Z) - Deep Weakly-supervised Anomaly Detection [118.55172352231381]
Pairwise Relation prediction Network (PReNet) learns pairwise relation features and anomaly scores.
PReNet can detect any seen/unseen abnormalities that fit the learned pairwise abnormal patterns.
Empirical results on 12 real-world datasets show that PReNet significantly outperforms nine competing methods in detecting seen and unseen anomalies.
arXiv Detail & Related papers (2019-10-30T00:40:25Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.