AIOps-Driven Enhancement of Log Anomaly Detection in Unsupervised Scenarios

• URL: http://arxiv.org/abs/2311.02621v1
• Date: Sun, 5 Nov 2023 11:16:24 GMT
• Title: AIOps-Driven Enhancement of Log Anomaly Detection in Unsupervised Scenarios
• Authors: Daksh Dave, Gauransh Sawhney, Dhruv Khut, Sahil Nawale, Pushkar Aggrawal, Prasenjit Bhavathankar
• Abstract summary: This study introduces a novel hybrid framework through an innovative algorithm that incorporates an unsupervised strategy. The proposed approach encompasses the utilization of both simulated and real-world datasets. The experimental results are highly promising, demonstrating significant reductions in pseudo-positives.
• Score: 0.18641315013048293
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Artificial intelligence operations (AIOps) play a pivotal role in identifying, mitigating, and analyzing anomalous system behaviors and alerts. However, the research landscape in this field remains limited, leaving significant gaps unexplored. This study introduces a novel hybrid framework through an innovative algorithm that incorporates an unsupervised strategy. This strategy integrates Principal Component Analysis (PCA) and Artificial Neural Networks (ANNs) and uses a custom loss function to substantially enhance the effectiveness of log anomaly detection. The proposed approach encompasses the utilization of both simulated and real-world datasets, including logs from SockShop and Hadoop Distributed File System (HDFS). The experimental results are highly promising, demonstrating significant reductions in pseudo-positives. Moreover, this strategy offers notable advantages, such as the ability to process logs in their raw, unprocessed form, and the potential for further enhancements. The successful implementation of this approach showcases a remarkable reduction in anomalous logs, thus unequivocally establishing the efficacy of the proposed methodology. Ultimately, this study makes a substantial contribution to the advancement of log anomaly detection within AIOps platforms, addressing the critical need for effective and efficient log analysis in modern and complex systems.

Related papers

• OMLog: Online Log Anomaly Detection for Evolving System with Meta-learning [10.181157278476428]
  OMLog is a real-time and reliable online log anomaly detection model. We introduce a maximum mean discrepancy-based distribution shift detection method. We also design an online learning mechanism based on meta-learning, which can effectively learn the highly repetitive patterns of log sequences.
  arXiv  Detail & Related papers  (2024-10-22T01:50:07Z)
• Geospatial Trajectory Generation via Efficient Abduction: Deployment for Independent Testing [1.8877926393541125]
  We show that we can abduce movement trajectories efficiently through an informed (i.e., A*) search. We also report on our own experiments showing that we not only provide exact results but also scale to very large scenarios.
  arXiv  Detail & Related papers  (2024-07-08T23:11:47Z)
• Deep Learning-based Anomaly Detection and Log Analysis for Computer Networks [5.809158072574843]
  We propose an innovative fusion model that integrates Isolation Forest, GAN, and Transformer. The model significantly improves the accuracy of anomaly detection while reducing the false alarm rate. It also performs well in the log analysis task and is able to quickly identify anomalous behaviors.
  arXiv  Detail & Related papers  (2024-07-08T06:07:51Z)
• Multi-modal Causal Structure Learning and Root Cause Analysis [67.67578590390907]
  We propose Mulan, a unified multi-modal causal structure learning method for root cause localization. We leverage a log-tailored language model to facilitate log representation learning, converting log sequences into time-series data. We also introduce a novel key performance indicator-aware attention mechanism for assessing modality reliability and co-learning a final causal graph.
  arXiv  Detail & Related papers  (2024-02-04T05:50:38Z)
• Progressing from Anomaly Detection to Automated Log Labeling and Pioneering Root Cause Analysis [53.24804865821692]
  This study introduces a taxonomy for log anomalies and explores automated data labeling to mitigate labeling challenges. The study envisions a future where root cause analysis follows anomaly detection, unraveling the underlying triggers of anomalies.
  arXiv  Detail & Related papers  (2023-12-22T15:04:20Z)
• GraphCloak: Safeguarding Task-specific Knowledge within Graph-structured Data from Unauthorized Exploitation [61.80017550099027]
  Graph Neural Networks (GNNs) are increasingly prevalent in a variety of fields. Growing concerns have emerged regarding the unauthorized utilization of personal data. Recent studies have shown that imperceptible poisoning attacks are an effective method of protecting image data from such misuse. This paper introduces GraphCloak to safeguard against the unauthorized usage of graph data.
  arXiv  Detail & Related papers  (2023-10-11T00:50:55Z)
• Combating Advanced Persistent Threats: Challenges and Solutions [20.81151411772311]
  The rise of advanced persistent threats (APTs) has marked a significant cybersecurity challenge. Provenance graph-based kernel-level auditing has emerged as a promising approach to enhance visibility and traceability. This paper proposes an efficient and robust APT defense scheme leveraging provenance graphs, including a network-level distributed audit model for cost-effective lateral attack reconstruction.
  arXiv  Detail & Related papers  (2023-09-18T05:46:11Z)
• REX: Rapid Exploration and eXploitation for AI Agents [103.68453326880456]
  We propose an enhanced approach for Rapid Exploration and eXploitation for AI Agents called REX. REX introduces an additional layer of rewards and integrates concepts similar to Upper Confidence Bound (UCB) scores, leading to more robust and efficient AI agent performance.
  arXiv  Detail & Related papers  (2023-07-18T04:26:33Z)
• Efficient selective attention LSTM for well log curve synthesis [0.0]
  This paper proposes a machine learning method that utilizes existing data to predict missing data. The proposed method builds on the traditional Long Short-Term Memory (LSTM) neural network by incorporating a self-attention mechanism. Experimental results demonstrate that the proposed method achieves higher accuracy compared to traditional curve synthesis methods.
  arXiv  Detail & Related papers  (2023-07-17T09:35:18Z)
• Energy-based Out-of-Distribution Detection for Graph Neural Networks [76.0242218180483]
  We propose a simple, powerful and efficient OOD detection model for GNN-based learning on graphs, which we call GNNSafe. GNNSafe achieves up to $17.0%$ AUROC improvement over state-of-the-arts and it could serve as simple yet strong baselines in such an under-developed area.
  arXiv  Detail & Related papers  (2023-02-06T16:38:43Z)
• DEALIO: Data-Efficient Adversarial Learning for Imitation from Observation [57.358212277226315]
  In imitation learning from observation IfO, a learning agent seeks to imitate a demonstrating agent using only observations of the demonstrated behavior without access to the control signals generated by the demonstrator. Recent methods based on adversarial imitation learning have led to state-of-the-art performance on IfO problems, but they typically suffer from high sample complexity due to a reliance on data-inefficient, model-free reinforcement learning algorithms. This issue makes them impractical to deploy in real-world settings, where gathering samples can incur high costs in terms of time, energy, and risk. We propose a more data-efficient IfO algorithm
  arXiv  Detail & Related papers  (2021-03-31T23:46:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.