LogLG: Weakly Supervised Log Anomaly Detection via Log-Event Graph
Construction
- URL: http://arxiv.org/abs/2208.10833v5
- Date: Tue, 11 Apr 2023 07:46:32 GMT
- Title: LogLG: Weakly Supervised Log Anomaly Detection via Log-Event Graph
Construction
- Authors: Hongcheng Guo, Yuhui Guo, Renjie Chen, Jian Yang, Jiaheng Liu, Zhoujun
Li, Tieqiao Zheng, Weichao Hou, Liangfan Zheng, Bo Zhang
- Abstract summary: We propose a novel weakly supervised log anomaly detection framework, named LogLG, to explore the semantic connections among keywords from sequences.
Specifically, we design an end-to-end iterative process, where the keywords of unlabeled logs are first extracted to construct a log-event graph.
Then, we build a subgraph annotator to generate pseudo labels for unlabeled log sequences.
- Score: 31.31712326361932
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Fully supervised log anomaly detection methods suffer the heavy burden of
annotating massive unlabeled log data. Recently, many semi-supervised methods
have been proposed to reduce annotation costs with the help of parsed
templates. However, these methods consider each keyword independently, which
disregards the correlation between keywords and the contextual relationships
among log sequences. In this paper, we propose a novel weakly supervised log
anomaly detection framework, named LogLG, to explore the semantic connections
among keywords from sequences. Specifically, we design an end-to-end iterative
process, where the keywords of unlabeled logs are first extracted to construct
a log-event graph. Then, we build a subgraph annotator to generate pseudo
labels for unlabeled log sequences. To ameliorate the annotation quality, we
adopt a self-supervised task to pre-train a subgraph annotator. After that, a
detection model is trained with the generated pseudo labels. Conditioned on the
classification results, we re-extract the keywords from the log sequences and
update the log-event graph for the next iteration. Experiments on five
benchmarks validate the effectiveness of LogLG for detecting anomalies on
unlabeled log data and demonstrate that LogLG, as the state-of-the-art weakly
supervised method, achieves significant performance improvements compared to
existing methods.
Related papers
- Log2graphs: An Unsupervised Framework for Log Anomaly Detection with Efficient Feature Extraction [1.474723404975345]
High cost of manual annotation and dynamic nature of usage scenarios present major challenges to effective log analysis.
This study proposes a novel log feature extraction model called DualGCN-LogAE, designed to adapt to various scenarios.
We also introduce Log2graphs, an unsupervised log anomaly detection method based on the feature extractor.
arXiv Detail & Related papers (2024-09-18T11:35:58Z) - Lemur: Log Parsing with Entropy Sampling and Chain-of-Thought Merging [33.522495018321386]
We introduce a cutting-edge textbfLog parsing framework with textbfEntropy sampling and Chain-of-Thought textbfMerging (Lemur)
We propose a novel sampling method inspired by information entropy, which efficiently clusters typical logs.
Lemur achieves the state-of-the-art performance and impressive efficiency.
arXiv Detail & Related papers (2024-02-28T09:51:55Z) - LogFormer: A Pre-train and Tuning Pipeline for Log Anomaly Detection [73.69399219776315]
We propose a unified Transformer-based framework for Log anomaly detection (LogFormer) to improve the generalization ability across different domains.
Specifically, our model is first pre-trained on the source domain to obtain shared semantic knowledge of log data.
Then, we transfer such knowledge to the target domain via shared parameters.
arXiv Detail & Related papers (2024-01-09T12:55:21Z) - GLAD: Content-aware Dynamic Graphs For Log Anomaly Detection [49.9884374409624]
GLAD is a Graph-based Log Anomaly Detection framework designed to detect anomalies in system logs.
We introduce GLAD, a Graph-based Log Anomaly Detection framework designed to detect anomalies in system logs.
arXiv Detail & Related papers (2023-09-12T04:21:30Z) - ClusterLog: Clustering Logs for Effective Log-based Anomaly Detection [3.3196401064045014]
This study proposes ClusterLog, a log pre-processing method that clusters the temporal sequence of log keys based on their semantic similarity.
By grouping semantically and sentimentally similar logs, this approach aims to represent log sequences with the smallest amount of unique log keys, intending to improve the ability of a downstream sequence-based model to effectively learn the log patterns.
arXiv Detail & Related papers (2023-01-19T01:54:48Z) - LogGD:Detecting Anomalies from System Logs by Graph Neural Networks [14.813971618949068]
We propose a novel graph-based log anomaly detection method, LogGD, to effectively address the issue.
We exploit the powerful capability of Graph Transformer Neural Network, which combines graph structure and node semantics for log-based anomaly detection.
arXiv Detail & Related papers (2022-09-16T11:51:58Z) - LogLAB: Attention-Based Labeling of Log Data Anomalies via Weak
Supervision [63.08516384181491]
We present LogLAB, a novel modeling approach for automated labeling of log messages without requiring manual work by experts.
Our method relies on estimated failure time windows provided by monitoring systems to produce precise labeled datasets in retrospect.
Our evaluation shows that LogLAB consistently outperforms nine benchmark approaches across three different datasets and maintains an F1-score of more than 0.98 even at large failure time windows.
arXiv Detail & Related papers (2021-11-02T15:16:08Z) - Robust and Transferable Anomaly Detection in Log Data using Pre-Trained
Language Models [59.04636530383049]
Anomalies or failures in large computer systems, such as the cloud, have an impact on a large number of users.
We propose a framework for anomaly detection in log data, as a major troubleshooting source of system information.
arXiv Detail & Related papers (2021-02-23T09:17:05Z) - Self-Attentive Classification-Based Anomaly Detection in Unstructured
Logs [59.04636530383049]
We propose Logsy, a classification-based method to learn log representations.
We show an average improvement of 0.25 in the F1 score, compared to the previous methods.
arXiv Detail & Related papers (2020-08-21T07:26:55Z) - Self-Supervised Log Parsing [59.04636530383049]
Large-scale software systems generate massive volumes of semi-structured log records.
Existing approaches rely on log-specifics or manual rule extraction.
We propose NuLog that utilizes a self-supervised learning model and formulates the parsing task as masked language modeling.
arXiv Detail & Related papers (2020-03-17T19:25:25Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.