ASGNet: Adaptive Semantic Gate Networks for Log-Based Anomaly Diagnosis

• URL: http://arxiv.org/abs/2402.11841v1
• Date: Mon, 19 Feb 2024 05:08:44 GMT
• Title: ASGNet: Adaptive Semantic Gate Networks for Log-Based Anomaly Diagnosis
• Authors: Haitian Yang, Degang Sun, Wen Liu, Yanshu Li, Yan Wang, Weiqing Huang
• Abstract summary: We propose an adaptive semantic gate networks (ASGNet) that combines statistical features and semantic features to consolidate log text semantic representation. ASGNet encodes statistical features via a variational encoding module and fuses useful information through a well-designed adaptive semantic threshold mechanism.
• Score: 6.399472066185473
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Logs are widely used in the development and maintenance of software systems. Logs can help engineers understand the runtime behavior of systems and diagnose system failures. For anomaly diagnosis, existing methods generally use log event data extracted from historical logs to build diagnostic models. However, we find that existing methods do not make full use of two types of features, (1) statistical features: some inherent statistical features in log data, such as word frequency and abnormal label distribution, are not well exploited. Compared with log raw data, statistical features are deterministic and naturally compatible with corresponding tasks. (2) semantic features: Logs contain the execution logic behind software systems, thus log statements share deep semantic relationships. How to effectively combine statistical features and semantic features in log data to improve the performance of log anomaly diagnosis is the key point of this paper. In this paper, we propose an adaptive semantic gate networks (ASGNet) that combines statistical features and semantic features to selectively use statistical features to consolidate log text semantic representation. Specifically, ASGNet encodes statistical features via a variational encoding module and fuses useful information through a well-designed adaptive semantic threshold mechanism. The threshold mechanism introduces the information flow into the classifier based on the confidence of the semantic features in the decision, which is conducive to training a robust classifier and can solve the overfitting problem caused by the use of statistical features. The experimental results on the real data set show that our method proposed is superior to all baseline methods in terms of various performance indicators.

Related papers

• A Hybrid Framework for Statistical Feature Selection and Image-Based Noise-Defect Detection [55.2480439325792]
  This paper presents a hybrid framework that integrates both statistical feature selection and classification techniques to improve defect detection accuracy. We present around 55 distinguished features that are extracted from industrial images, which are then analyzed using statistical methods. By integrating these methods with flexible machine learning applications, the proposed framework improves detection accuracy and reduces false positives and misclassifications.
  arXiv  Detail & Related papers  (2024-12-11T22:12:21Z)
• Log2graphs: An Unsupervised Framework for Log Anomaly Detection with Efficient Feature Extraction [1.474723404975345]
  High cost of manual annotation and dynamic nature of usage scenarios present major challenges to effective log analysis. This study proposes a novel log feature extraction model called DualGCN-LogAE, designed to adapt to various scenarios. We also introduce Log2graphs, an unsupervised log anomaly detection method based on the feature extractor.
  arXiv  Detail & Related papers  (2024-09-18T11:35:58Z)
• LogFormer: A Pre-train and Tuning Pipeline for Log Anomaly Detection [73.69399219776315]
  We propose a unified Transformer-based framework for Log anomaly detection (LogFormer) to improve the generalization ability across different domains. Specifically, our model is first pre-trained on the source domain to obtain shared semantic knowledge of log data. Then, we transfer such knowledge to the target domain via shared parameters.
  arXiv  Detail & Related papers  (2024-01-09T12:55:21Z)
• GLAD: Content-aware Dynamic Graphs For Log Anomaly Detection [49.9884374409624]
  GLAD is a Graph-based Log Anomaly Detection framework designed to detect anomalies in system logs. We introduce GLAD, a Graph-based Log Anomaly Detection framework designed to detect anomalies in system logs.
  arXiv  Detail & Related papers  (2023-09-12T04:21:30Z)
• PULL: Reactive Log Anomaly Detection Based On Iterative PU Learning [58.85063149619348]
  We propose PULL, an iterative log analysis method for reactive anomaly detection based on estimated failure time windows. Our evaluation shows that PULL consistently outperforms ten benchmark baselines across three different datasets.
  arXiv  Detail & Related papers  (2023-01-25T16:34:43Z)
• LogGD:Detecting Anomalies from System Logs by Graph Neural Networks [14.813971618949068]
  We propose a novel graph-based log anomaly detection method, LogGD, to effectively address the issue. We exploit the powerful capability of Graph Transformer Neural Network, which combines graph structure and node semantics for log-based anomaly detection.
  arXiv  Detail & Related papers  (2022-09-16T11:51:58Z)
• Leveraging Log Instructions in Log-based Anomaly Detection [0.5949779668853554]
  We propose a method for reliable and practical anomaly detection from system logs. It overcomes the common disadvantage of related works by building an anomaly detection model with log instructions from the source code of 1000+ GitHub projects. The proposed method, named ADLILog, combines the log instructions and the data from the system of interest (target system) to learn a deep neural network model.
  arXiv  Detail & Related papers  (2022-07-07T10:22:10Z)
• Log-based Anomaly Detection Without Log Parsing [7.66638994053231]
  We propose NeuralLog, a novel log-based anomaly detection approach that does not require log parsing. Our experimental results show that the proposed approach can effectively understand the semantic meaning of log messages. Overall, NeuralLog achieves F1-scores greater than 0.95 on four public datasets, outperforming the existing approaches.
  arXiv  Detail & Related papers  (2021-08-04T10:42:13Z)
• Robust and Transferable Anomaly Detection in Log Data using Pre-Trained Language Models [59.04636530383049]
  Anomalies or failures in large computer systems, such as the cloud, have an impact on a large number of users. We propose a framework for anomaly detection in log data, as a major troubleshooting source of system information.
  arXiv  Detail & Related papers  (2021-02-23T09:17:05Z)
• Self-Attentive Classification-Based Anomaly Detection in Unstructured Logs [59.04636530383049]
  We propose Logsy, a classification-based method to learn log representations. We show an average improvement of 0.25 in the F1 score, compared to the previous methods.
  arXiv  Detail & Related papers  (2020-08-21T07:26:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.