Deep Learning for Anomaly Detection in Log Data: A Survey

• URL: http://arxiv.org/abs/2207.03820v2
• Date: Mon, 15 May 2023 10:16:10 GMT
• Title: Deep Learning for Anomaly Detection in Log Data: A Survey
• Authors: Max Landauer, Sebastian Onder, Florian Skopik, Markus Wurzenberger
• Abstract summary: Self-learning anomaly detection techniques capture patterns in log data and report unexpected log event occurrences. Deep learning neural networks for this purpose have been presented. There exist many different architectures for deep learning and it is non-trivial to encode raw and unstructured log data.
• Score: 3.508620069426877
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Automatic log file analysis enables early detection of relevant incidents such as system failures. In particular, self-learning anomaly detection techniques capture patterns in log data and subsequently report unexpected log event occurrences to system operators without the need to provide or manually model anomalous scenarios in advance. Recently, an increasing number of approaches leveraging deep learning neural networks for this purpose have been presented. These approaches have demonstrated superior detection performance in comparison to conventional machine learning techniques and simultaneously resolve issues with unstable data formats. However, there exist many different architectures for deep learning and it is non-trivial to encode raw and unstructured log data to be analyzed by neural networks. We therefore carry out a systematic literature review that provides an overview of deployed models, data pre-processing mechanisms, anomaly detection techniques, and evaluations. The survey does not quantitatively compare existing approaches but instead aims to help readers understand relevant aspects of different model architectures and emphasizes open issues for future work.

Related papers

• Can Tree Based Approaches Surpass Deep Learning in Anomaly Detection? A Benchmarking Study [0.6291443816903801]
  This paper evaluates a diverse array of machine learning-based anomaly detection algorithms. The paper contributes significantly by conducting an unbiased comparison of various anomaly detection algorithms.
  arXiv  Detail & Related papers  (2024-02-11T19:12:51Z)
• Log-based Anomaly Detection of Enterprise Software: An Empirical Study [0.0]
  We evaluate several state-of-the-art anomaly detection models on an industrial dataset from our research partner. Results show that while all models are capable of detecting anomalies, certain models are better suited for less-structured datasets.
  arXiv  Detail & Related papers  (2023-10-31T14:32:08Z)
• Deep networks for system identification: a Survey [56.34005280792013]
  System identification learns mathematical descriptions of dynamic systems from input-output data. Main aim of the identified model is to predict new data from previous observations. We discuss architectures commonly adopted in the literature, like feedforward, convolutional, and recurrent networks.
  arXiv  Detail & Related papers  (2023-01-30T12:38:31Z)
• PULL: Reactive Log Anomaly Detection Based On Iterative PU Learning [58.85063149619348]
  We propose PULL, an iterative log analysis method for reactive anomaly detection based on estimated failure time windows. Our evaluation shows that PULL consistently outperforms ten benchmark baselines across three different datasets.
  arXiv  Detail & Related papers  (2023-01-25T16:34:43Z)
• Log-based Anomaly Detection with Deep Learning: How Far Are We? [7.967230034960396]
  We conduct an in-depth analysis of five state-of-the-art deep learning-based models for detecting system anomalies on four public log datasets. Our results point out that all the studied models do not always work well.
  arXiv  Detail & Related papers  (2022-02-09T06:27:11Z)
• A Meta-learning Approach to Reservoir Computing: Time Series Prediction with Limited Data [0.0]
  We present a data-driven approach to automatically extract an appropriate model structure from experimentally observed processes. We demonstrate our approach on a simple benchmark problem, where it beats the state of the art meta-learning techniques.
  arXiv  Detail & Related papers  (2021-10-07T18:23:14Z)
• Anomaly Detection on Attributed Networks via Contrastive Self-Supervised Learning [50.24174211654775]
  We present a novel contrastive self-supervised learning framework for anomaly detection on attributed networks. Our framework fully exploits the local information from network data by sampling a novel type of contrastive instance pair. A graph neural network-based contrastive learning model is proposed to learn informative embedding from high-dimensional attributes and local structure.
  arXiv  Detail & Related papers  (2021-02-27T03:17:20Z)
• Robust and Transferable Anomaly Detection in Log Data using Pre-Trained Language Models [59.04636530383049]
  Anomalies or failures in large computer systems, such as the cloud, have an impact on a large number of users. We propose a framework for anomaly detection in log data, as a major troubleshooting source of system information.
  arXiv  Detail & Related papers  (2021-02-23T09:17:05Z)
• Model-Based Deep Learning [155.063817656602]
  Signal processing, communications, and control have traditionally relied on classical statistical modeling techniques. Deep neural networks (DNNs) use generic architectures which learn to operate from data, and demonstrate excellent performance. We are interested in hybrid techniques that combine principled mathematical models with data-driven systems to benefit from the advantages of both approaches.
  arXiv  Detail & Related papers  (2020-12-15T16:29:49Z)
• Self-Attentive Classification-Based Anomaly Detection in Unstructured Logs [59.04636530383049]
  We propose Logsy, a classification-based method to learn log representations. We show an average improvement of 0.25 in the F1 score, compared to the previous methods.
  arXiv  Detail & Related papers  (2020-08-21T07:26:55Z)
• Complex Sequential Data Analysis: A Systematic Literature Review of Existing Algorithms [0.9649642656207869]
  This paper reviews past approaches to the use of deep-learning frameworks for the analysis of irregular-patterned datasets. Traditional deep-learning methods perform poorly or even fail when trying to analyse these datasets. The performance of deep-learning frameworks was found to be evaluated mainly using mean absolute error and root mean square error accuracy metrics.
  arXiv  Detail & Related papers  (2020-07-22T17:53:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.