A Taxonomy of Anomalies in Log Data
- URL: http://arxiv.org/abs/2111.13462v1
- Date: Fri, 26 Nov 2021 12:23:06 GMT
- Title: A Taxonomy of Anomalies in Log Data
- Authors: Thorsten Wittkopp, Philipp Wiesner, Dominik Scheinert, Odej Kao
- Abstract summary: A common taxonomy for anomalies already exists, but it has not yet been applied specifically to log data.
We present a taxonomy for different kinds of log data anomalies and introduce a method for analyzing such anomalies in labeled datasets.
Our results show, that the most common anomaly type is also the easiest to predict.
- Score: 0.09558392439655014
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Log data anomaly detection is a core component in the area of artificial
intelligence for IT operations. However, the large amount of existing methods
makes it hard to choose the right approach for a specific system. A better
understanding of different kinds of anomalies, and which algorithms are
suitable for detecting them, would support researchers and IT operators.
Although a common taxonomy for anomalies already exists, it has not yet been
applied specifically to log data, pointing out the characteristics and
peculiarities in this domain.
In this paper, we present a taxonomy for different kinds of log data
anomalies and introduce a method for analyzing such anomalies in labeled
datasets. We applied our taxonomy to the three common benchmark datasets
Thunderbird, Spirit, and BGL, and trained five state-of-the-art unsupervised
anomaly detection algorithms to evaluate their performance in detecting
different kinds of anomalies. Our results show, that the most common anomaly
type is also the easiest to predict. Moreover, deep learning-based approaches
outperform data mining-based approaches in all anomaly types, but especially
when it comes to detecting contextual anomalies.
Related papers
- ARC: A Generalist Graph Anomaly Detector with In-Context Learning [62.202323209244]
ARC is a generalist GAD approach that enables a one-for-all'' GAD model to detect anomalies across various graph datasets on-the-fly.
equipped with in-context learning, ARC can directly extract dataset-specific patterns from the target dataset.
Extensive experiments on multiple benchmark datasets from various domains demonstrate the superior anomaly detection performance, efficiency, and generalizability of ARC.
arXiv Detail & Related papers (2024-05-27T02:42:33Z) - Anomaly Detection in Graph Structured Data: A Survey [0.46040036610482665]
We discuss a comprehensive overview of anomaly detection techniques on graph data.
We present a new taxonomy that categorizes the different state-of-the-art anomaly detection methods.
arXiv Detail & Related papers (2024-05-10T01:30:25Z) - Progressing from Anomaly Detection to Automated Log Labeling and
Pioneering Root Cause Analysis [53.24804865821692]
This study introduces a taxonomy for log anomalies and explores automated data labeling to mitigate labeling challenges.
The study envisions a future where root cause analysis follows anomaly detection, unraveling the underlying triggers of anomalies.
arXiv Detail & Related papers (2023-12-22T15:04:20Z) - A Critical Review of Common Log Data Sets Used for Evaluation of
Sequence-based Anomaly Detection Techniques [2.5339493426758906]
We analyze six publicly available log data sets with focus on the manifestations of anomalies and simple techniques for their detection.
Our findings suggest that most anomalies are not directly related to sequential manifestations and that advanced detection techniques are not required to achieve high detection rates on these data sets.
arXiv Detail & Related papers (2023-09-06T09:31:17Z) - WePaMaDM-Outlier Detection: Weighted Outlier Detection using Pattern
Approaches for Mass Data Mining [0.6754597324022876]
Outlier detection can reveal vital information about system faults, fraudulent activities, and patterns in the data.
This article proposed the WePaMaDM-Outlier Detection with distinct mass data mining domain.
It also investigates the significance of data modeling in outlier detection techniques in surveillance, fault detection, and trend analysis.
arXiv Detail & Related papers (2023-06-09T07:00:00Z) - Explainable Deep Few-shot Anomaly Detection with Deviation Networks [123.46611927225963]
We introduce a novel weakly-supervised anomaly detection framework to train detection models.
The proposed approach learns discriminative normality by leveraging the labeled anomalies and a prior probability.
Our model is substantially more sample-efficient and robust, and performs significantly better than state-of-the-art competing methods in both closed-set and open-set settings.
arXiv Detail & Related papers (2021-08-01T14:33:17Z) - A Typology of Data Anomalies [0.0]
Anomalies are cases that are in some way unusual and do not appear to fit the general patterns present in the dataset.
This paper introduces a general typology of anomalies that offers a clear and tangible definition of the different types of anomalies in datasets.
arXiv Detail & Related papers (2021-07-04T13:12:24Z) - Toward Deep Supervised Anomaly Detection: Reinforcement Learning from
Partially Labeled Anomaly Data [150.9270911031327]
We consider the problem of anomaly detection with a small set of partially labeled anomaly examples and a large-scale unlabeled dataset.
Existing related methods either exclusively fit the limited anomaly examples that typically do not span the entire set of anomalies, or proceed with unsupervised learning from the unlabeled data.
We propose here instead a deep reinforcement learning-based approach that enables an end-to-end optimization of the detection of both labeled and unlabeled anomalies.
arXiv Detail & Related papers (2020-09-15T03:05:39Z) - Self-Attentive Classification-Based Anomaly Detection in Unstructured
Logs [59.04636530383049]
We propose Logsy, a classification-based method to learn log representations.
We show an average improvement of 0.25 in the F1 score, compared to the previous methods.
arXiv Detail & Related papers (2020-08-21T07:26:55Z) - Deep Weakly-supervised Anomaly Detection [118.55172352231381]
Pairwise Relation prediction Network (PReNet) learns pairwise relation features and anomaly scores.
PReNet can detect any seen/unseen abnormalities that fit the learned pairwise abnormal patterns.
Empirical results on 12 real-world datasets show that PReNet significantly outperforms nine competing methods in detecting seen and unseen anomalies.
arXiv Detail & Related papers (2019-10-30T00:40:25Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.