MoniLog: An Automated Log-Based Anomaly Detection System for Cloud Computing Infrastructures

• URL: http://arxiv.org/abs/2304.11940v1
• Date: Mon, 24 Apr 2023 09:21:52 GMT
• Title: MoniLog: An Automated Log-Based Anomaly Detection System for Cloud Computing Infrastructures
• Authors: Arthur Vervaet
• Abstract summary: MoniLog is a distributed approach to detect real-time anomalies within large-scale environments. It aims to detect sequential and quantitative anomalies within a multi-source log stream.
• Score: 3.04585143845864
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Within today's large-scale systems, one anomaly can impact millions of users. Detecting such events in real-time is essential to maintain the quality of services. It allows the monitoring team to prevent or diminish the impact of a failure. Logs are a core part of software development and maintenance, by recording detailed information at runtime. Such log data are universally available in nearly all computer systems. They enable developers as well as system maintainers to monitor and dissect anomalous events. For Cloud computing companies and large online platforms in general, growth is linked to the scaling potential. Automatizing the anomaly detection process is a promising way to ensure the scalability of monitoring capacities regarding the increasing volume of logs generated by modern systems. In this paper, we will introduce MoniLog, a distributed approach to detect real-time anomalies within large-scale environments. It aims to detect sequential and quantitative anomalies within a multi-source log stream. MoniLog is designed to structure a log stream and perform the monitoring of anomalous sequences. Its output classifier learns from the administrator's actions to label and evaluate the criticality level of anomalies.

Related papers

• Log Summarisation for Defect Evolution Analysis [14.055261850785456]
  We suggest an online semantic-based clustering approach to error logs. We also introduce a novel metric to evaluate the performance of temporal log clusters.
  arXiv  Detail & Related papers  (2024-03-13T09:18:46Z)
• GLAD: Content-aware Dynamic Graphs For Log Anomaly Detection [49.9884374409624]
  GLAD is a Graph-based Log Anomaly Detection framework designed to detect anomalies in system logs. We introduce GLAD, a Graph-based Log Anomaly Detection framework designed to detect anomalies in system logs.
  arXiv  Detail & Related papers  (2023-09-12T04:21:30Z)
• EvLog: Identifying Anomalous Logs over Software Evolution [31.46106509190191]
  We propose a novel unsupervised approach named Evolving Log extractor (EvLog) to process logs without parsing. EvLog implements an anomaly discriminator with an attention mechanism to identify the anomalous logs and avoid the issue brought by the unstable sequence. EvLog has shown effectiveness in two real-world system evolution log datasets with an average F1 score of 0.955 and 0.847 in the intra-version setting and inter-version setting, respectively.
  arXiv  Detail & Related papers  (2023-06-02T12:58:00Z)
• PULL: Reactive Log Anomaly Detection Based On Iterative PU Learning [58.85063149619348]
  We propose PULL, an iterative log analysis method for reactive anomaly detection based on estimated failure time windows. Our evaluation shows that PULL consistently outperforms ten benchmark baselines across three different datasets.
  arXiv  Detail & Related papers  (2023-01-25T16:34:43Z)
• LogGD:Detecting Anomalies from System Logs by Graph Neural Networks [14.813971618949068]
  We propose a novel graph-based log anomaly detection method, LogGD, to effectively address the issue. We exploit the powerful capability of Graph Transformer Neural Network, which combines graph structure and node semantics for log-based anomaly detection.
  arXiv  Detail & Related papers  (2022-09-16T11:51:58Z)
• syslrn: Learning What to Monitor for Efficient Anomaly Detection [3.071931695335886]
  syslrn is a system that first builds an understanding of a target system offline, and then tailors the online monitoring instrumentation based on the learned identifiers of normal behavior. We show in a case study for the monitoring of failures that it can outperform state-of-the-art log-analysis systems with little overhead.
  arXiv  Detail & Related papers  (2022-03-29T08:10:06Z)
• Online Self-Evolving Anomaly Detection in Cloud Computing Environments [6.480575492140354]
  We present a emphself-evolving anomaly detection (SEAD) framework for cloud dependability assurance. Our framework self-evolves by exploring newly verified anomaly records and continuously updating the anomaly detector online. Our detectors can achieve 88.94% in sensitivity and 94.60% on average, which makes them suitable for real-world deployment.
  arXiv  Detail & Related papers  (2021-11-16T05:13:38Z)
• LogLAB: Attention-Based Labeling of Log Data Anomalies via Weak Supervision [63.08516384181491]
  We present LogLAB, a novel modeling approach for automated labeling of log messages without requiring manual work by experts. Our method relies on estimated failure time windows provided by monitoring systems to produce precise labeled datasets in retrospect. Our evaluation shows that LogLAB consistently outperforms nine benchmark approaches across three different datasets and maintains an F1-score of more than 0.98 even at large failure time windows.
  arXiv  Detail & Related papers  (2021-11-02T15:16:08Z)
• Robust and Transferable Anomaly Detection in Log Data using Pre-Trained Language Models [59.04636530383049]
  Anomalies or failures in large computer systems, such as the cloud, have an impact on a large number of users. We propose a framework for anomaly detection in log data, as a major troubleshooting source of system information.
  arXiv  Detail & Related papers  (2021-02-23T09:17:05Z)
• Self-Attentive Classification-Based Anomaly Detection in Unstructured Logs [59.04636530383049]
  We propose Logsy, a classification-based method to learn log representations. We show an average improvement of 0.25 in the F1 score, compared to the previous methods.
  arXiv  Detail & Related papers  (2020-08-21T07:26:55Z)
• Self-Supervised Log Parsing [59.04636530383049]
  Large-scale software systems generate massive volumes of semi-structured log records. Existing approaches rely on log-specifics or manual rule extraction. We propose NuLog that utilizes a self-supervised learning model and formulates the parsing task as masked language modeling.
  arXiv  Detail & Related papers  (2020-03-17T19:25:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.