Using a Neural Network to Detect Anomalies given an N-gram Profile
- URL: http://arxiv.org/abs/2104.05571v1
- Date: Mon, 12 Apr 2021 15:40:43 GMT
- Title: Using a Neural Network to Detect Anomalies given an N-gram Profile
- Authors: Byunggu Yu, Junwhan Kim
- Abstract summary: Anomaly detection is designed to profile the normal runtime behavior of computer programs.
Normal but unobserved behavior can trigger false positives.
This paper presents our study on how to explain the presence of anomalies using a neural network.
- Score: 0.0
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: In order to detect unknown intrusions and runtime errors of computer
programs, the cyber-security community has developed various detection
techniques. Anomaly detection is an approach that is designed to profile the
normal runtime behavior of computer programs in order to detect intrusions and
errors as anomalous deviations from the observed normal. However, normal but
unobserved behavior can trigger false positives. This limitation has
significantly decreased the practical viability of anomaly detection
techniques. Reported approaches to this limitation span a simple alert
threshold definition to distribution models for approximating all normal
behavior based on the limited observation. However, each assumption or
approximation poses the potential for even greater false positive rates. This
paper presents our study on how to explain the presence of anomalies using a
neural network, particularly Long Short-Term Memory, independent of actual data
distributions. We present and compare three anomaly detection models, and
report on our experience running different types of attacks on an Apache
Hypertext Transfer Protocol server. We performed a comparative study, focusing
on each model's ability to detect the onset of each attack while avoiding false
positives resulting from unknown normal behavior. Our best-performing model
detected the true onset of every attack with zero false positives.
Related papers
- Can I trust my anomaly detection system? A case study based on explainable AI [0.4416503115535552]
This case study explores the robustness of an anomaly detection system based on variational autoencoder generative models.
The goal is to get a different perspective on the real performances of anomaly detectors that use reconstruction differences.
arXiv Detail & Related papers (2024-07-29T12:39:07Z) - CARLA: Self-supervised Contrastive Representation Learning for Time Series Anomaly Detection [53.83593870825628]
One main challenge in time series anomaly detection (TSAD) is the lack of labelled data in many real-life scenarios.
Most of the existing anomaly detection methods focus on learning the normal behaviour of unlabelled time series in an unsupervised manner.
We introduce a novel end-to-end self-supervised ContrAstive Representation Learning approach for time series anomaly detection.
arXiv Detail & Related papers (2023-08-18T04:45:56Z) - Are we certain it's anomalous? [57.729669157989235]
Anomaly detection in time series is a complex task since anomalies are rare due to highly non-linear temporal correlations.
Here we propose the novel use of Hyperbolic uncertainty for Anomaly Detection (HypAD)
HypAD learns self-supervisedly to reconstruct the input signal.
arXiv Detail & Related papers (2022-11-16T21:31:39Z) - Catching Both Gray and Black Swans: Open-set Supervised Anomaly
Detection [90.32910087103744]
A few labeled anomaly examples are often available in many real-world applications.
These anomaly examples provide valuable knowledge about the application-specific abnormality.
Those anomalies seen during training often do not illustrate every possible class of anomaly.
This paper tackles open-set supervised anomaly detection.
arXiv Detail & Related papers (2022-03-28T05:21:37Z) - Explainable Deep Few-shot Anomaly Detection with Deviation Networks [123.46611927225963]
We introduce a novel weakly-supervised anomaly detection framework to train detection models.
The proposed approach learns discriminative normality by leveraging the labeled anomalies and a prior probability.
Our model is substantially more sample-efficient and robust, and performs significantly better than state-of-the-art competing methods in both closed-set and open-set settings.
arXiv Detail & Related papers (2021-08-01T14:33:17Z) - Understanding the Effect of Bias in Deep Anomaly Detection [15.83398707988473]
Anomaly detection presents a unique challenge in machine learning, due to the scarcity of labeled anomaly data.
Recent work attempts to mitigate such problems by augmenting training of deep anomaly detection models with additional labeled anomaly samples.
In this paper, we aim to understand the effect of a biased anomaly set on anomaly detection.
arXiv Detail & Related papers (2021-05-16T03:55:02Z) - TadGAN: Time Series Anomaly Detection Using Generative Adversarial
Networks [73.01104041298031]
TadGAN is an unsupervised anomaly detection approach built on Generative Adversarial Networks (GANs)
To capture the temporal correlations of time series, we use LSTM Recurrent Neural Networks as base models for Generators and Critics.
To demonstrate the performance and generalizability of our approach, we test several anomaly scoring techniques and report the best-suited one.
arXiv Detail & Related papers (2020-09-16T15:52:04Z) - $\text{A}^3$: Activation Anomaly Analysis [0.7734726150561088]
We show that the hidden activation values contain information useful to distinguish between normal and anomalous samples.
Our approach combines three neural networks in a purely data-driven end-to-end model.
Thanks to the anomaly network, our method even works in strict semi-supervised settings.
arXiv Detail & Related papers (2020-03-03T21:23:56Z) - Deep Weakly-supervised Anomaly Detection [118.55172352231381]
Pairwise Relation prediction Network (PReNet) learns pairwise relation features and anomaly scores.
PReNet can detect any seen/unseen abnormalities that fit the learned pairwise abnormal patterns.
Empirical results on 12 real-world datasets show that PReNet significantly outperforms nine competing methods in detecting seen and unseen anomalies.
arXiv Detail & Related papers (2019-10-30T00:40:25Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.