Improving log-based anomaly detection through learned adaptive filter

• URL: http://arxiv.org/abs/2504.02994v1
• Date: Thu, 03 Apr 2025 19:31:24 GMT
• Title: Improving log-based anomaly detection through learned adaptive filter
• Authors: Yiyuan Xiong, Shaofeng Cai,
• Abstract summary: State-of-the-art unsupervised methods predict the next log event given a log sequence and apply fixed configurations that use the same filter condition.<n>Deep reinforcement learning (DRL) are widely applied to make intelligent decisions in a dynamic environment.<n>We propose a novel approach based on DRL to construct a learned adaptive filter and apply different normal/abnormal filter thresholds for different log sequences.
• Score: 4.0534963661486385
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Log messages record important system runtime information and are useful for detecting anomalous behaviors and managing modern software systems. Many supervised and unsupervised learning methods have been proposed recently for log-based anomaly detection. State-of-the-art unsupervised methods predict the next log event given a log sequence and apply fixed configurations that use the same filter condition (i.e. k, the top k predicted log events will be regarded as normal next events) which leads to inferior performance in the detection stage because it sets one fixed k for all log sequences, which ignores the dynamic nature and variance in different log sequences. Recently, deep reinforcement learning (DRL) are widely applied to make intelligent decisions in a dynamic environment. In this work, we contend that it is necessary to apply adaptive filters for different log sequences. To achieve this, we propose a novel approach based on DRL to construct a learned adaptive filter and apply different normal/abnormal filter thresholds for different log sequences. We define the Markov Decision Process (MDP) and formulate the learned adaptive filter as a problem that can be solved by DRL. We evaluate the learned adaptive filter on two state-of-the-art log-based anomaly detection unsupervised approaches DeepLog and LogAnomaly in two datasets HDFS and BGL. Extensive experiments show that our approach outperforms the fixed configurations and achieves significantly better performance in log-based anomaly detection.

Related papers

• OMLog: Online Log Anomaly Detection for Evolving System with Meta-learning [10.181157278476428]
  OMLog is a real-time and reliable online log anomaly detection model. We introduce a maximum mean discrepancy-based distribution shift detection method. We also design an online learning mechanism based on meta-learning, which can effectively learn the highly repetitive patterns of log sequences.
  arXiv  Detail & Related papers  (2024-10-22T01:50:07Z)
• FastLogAD: Log Anomaly Detection with Mask-Guided Pseudo Anomaly Generation and Discrimination [13.458633961243498]
  We propose FastLogAD, a generator-discriminator framework trained to generate pseudo-abnormal logs. During the discriminative stage, FastLogAD learns a distinct separation between normal and pseudoabnormal samples. Compared to previous methods, FastLogAD achieves at least x10 speed increase in anomaly detection.
  arXiv  Detail & Related papers  (2024-04-12T18:23:29Z)
• LogFormer: A Pre-train and Tuning Pipeline for Log Anomaly Detection [73.69399219776315]
  We propose a unified Transformer-based framework for Log anomaly detection (LogFormer) to improve the generalization ability across different domains. Specifically, our model is first pre-trained on the source domain to obtain shared semantic knowledge of log data. Then, we transfer such knowledge to the target domain via shared parameters.
  arXiv  Detail & Related papers  (2024-01-09T12:55:21Z)
• RAPID: Training-free Retrieval-based Log Anomaly Detection with PLM considering Token-level information [7.861095039299132]
  The need for log anomaly detection is growing, especially in real-world applications. Traditional deep learning-based anomaly detection models require dataset-specific training, leading to corresponding delays. We introduce RAPID, a model that capitalizes on the inherent features of log data to enable anomaly detection without training delays.
  arXiv  Detail & Related papers  (2023-11-09T06:11:44Z)
• GLAD: Content-aware Dynamic Graphs For Log Anomaly Detection [49.9884374409624]
  GLAD is a Graph-based Log Anomaly Detection framework designed to detect anomalies in system logs. We introduce GLAD, a Graph-based Log Anomaly Detection framework designed to detect anomalies in system logs.
  arXiv  Detail & Related papers  (2023-09-12T04:21:30Z)
• PULL: Reactive Log Anomaly Detection Based On Iterative PU Learning [58.85063149619348]
  We propose PULL, an iterative log analysis method for reactive anomaly detection based on estimated failure time windows. Our evaluation shows that PULL consistently outperforms ten benchmark baselines across three different datasets.
  arXiv  Detail & Related papers  (2023-01-25T16:34:43Z)
• LogGD:Detecting Anomalies from System Logs by Graph Neural Networks [14.813971618949068]
  We propose a novel graph-based log anomaly detection method, LogGD, to effectively address the issue. We exploit the powerful capability of Graph Transformer Neural Network, which combines graph structure and node semantics for log-based anomaly detection.
  arXiv  Detail & Related papers  (2022-09-16T11:51:58Z)
• Filter-enhanced MLP is All You Need for Sequential Recommendation [89.0974365344997]
  In online platforms, logged user behavior data is inevitable to contain noise. We borrow the idea of filtering algorithms from signal processing that attenuates the noise in the frequency domain. We propose textbfFMLP-Rec, an all-MLP model with learnable filters for sequential recommendation task.
  arXiv  Detail & Related papers  (2022-02-28T05:49:35Z)
• A2Log: Attentive Augmented Log Anomaly Detection [53.06341151551106]
  Anomaly detection becomes increasingly important for the dependability and serviceability of IT services. Existing unsupervised methods need anomaly examples to obtain a suitable decision boundary. We develop A2Log, which is an unsupervised anomaly detection method consisting of two steps: Anomaly scoring and anomaly decision.
  arXiv  Detail & Related papers  (2021-09-20T13:40:21Z)
• Self-Attentive Classification-Based Anomaly Detection in Unstructured Logs [59.04636530383049]
  We propose Logsy, a classification-based method to learn log representations. We show an average improvement of 0.25 in the F1 score, compared to the previous methods.
  arXiv  Detail & Related papers  (2020-08-21T07:26:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.