Distributed Log-driven Anomaly Detection System based on Evolving Decision Making

• URL: http://arxiv.org/abs/2504.02322v1
• Date: Thu, 03 Apr 2025 06:50:30 GMT
• Title: Distributed Log-driven Anomaly Detection System based on Evolving Decision Making
• Authors: Zhuoran Tan, Qiyuan Wang, Christos Anagnostopoulos, Shameem P. Parambath, Jeremy Singer, Sam Temple,
• Abstract summary: CEDLog is a framework that implements distributed computing for scalable processing by integrating Apache Airflow and Dask.<n>In CEDLog, anomalies are detected through the synthesis of Multi-layer Perceptron (MLP) and Graph Convolutional Networks (GCNs) using critical features present in event logs.
• Score: 4.183506125389502
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Effective anomaly detection from logs is crucial for enhancing cybersecurity defenses by enabling the early identification of threats. Despite advances in anomaly detection, existing systems often fall short in areas such as post-detection validation, scalability, and effective maintenance. These limitations not only hinder the detection of new threats but also impair overall system performance. To address these challenges, we propose CEDLog, a novel practical framework that integrates Elastic Weight Consolidation (EWC) for continual learning and implements distributed computing for scalable processing by integrating Apache Airflow and Dask. In CEDLog, anomalies are detected through the synthesis of Multi-layer Perceptron (MLP) and Graph Convolutional Networks (GCNs) using critical features present in event logs. Through comparisons with update strategies on large-scale datasets, we demonstrate the strengths of CEDLog, showcasing efficient updates and low false positives

Related papers

• Decentralized Entropy-Driven Ransomware Detection Using Autonomous Neural Graph Embeddings [0.0]
  The framework operates on a distributed network of nodes, eliminating single points of failure and enhancing resilience against targeted attacks.<n>The integration of graph-based modeling and machine learning techniques enables the framework to capture complex system interactions.<n>Case studies validate its effectiveness in real-world scenarios, showcasing its ability to detect and mitigate ransomware attacks within minutes of their initiation.
  arXiv  Detail & Related papers  (2025-02-11T11:59:10Z)
• iCNN-LSTM: A batch-based incremental ransomware detection system using Sysmon [1.495391051525033]
  This study presents a novel detection system that integrates Convolutional Neural Networks (CNNs) and Long Short-Term Memory (LSTM) networks.<n>By leveraging Sysmon logs, the system enables real-time analysis on Windows-based endpoints.
  arXiv  Detail & Related papers  (2025-01-02T05:57:41Z)
• Learning in Multiple Spaces: Few-Shot Network Attack Detection with Metric-Fused Prototypical Networks [47.18575262588692]
  We propose a novel Multi-Space Prototypical Learning framework tailored for few-shot attack detection.<n>By leveraging Polyak-averaged prototype generation, the framework stabilizes the learning process and effectively adapts to rare and zero-day attacks.<n> Experimental results on benchmark datasets demonstrate that MSPL outperforms traditional approaches in detecting low-profile and novel attack types.
  arXiv  Detail & Related papers  (2024-12-28T00:09:46Z)
• SCADE: Scalable Framework for Anomaly Detection in High-Performance System [0.0]
  Command-line interfaces remain integral to high-performance computing environments.<n>Traditional security solutions struggle to detect anomalies due to their context-specific nature, lack of labeled data, and the prevalence of sophisticated attacks like Living-off-the-Land (LOL)<n>We introduce the Scalable Command-Line Anomaly Detection Engine (SCADE), a framework that combines global statistical models with local context-specific analysis for unsupervised anomaly detection.
  arXiv  Detail & Related papers  (2024-12-05T15:39:13Z)
• OMLog: Online Log Anomaly Detection for Evolving System with Meta-learning [10.181157278476428]
  OMLog is a real-time and reliable online log anomaly detection model. We introduce a maximum mean discrepancy-based distribution shift detection method. We also design an online learning mechanism based on meta-learning, which can effectively learn the highly repetitive patterns of log sequences.
  arXiv  Detail & Related papers  (2024-10-22T01:50:07Z)
• Convolutional Neural Network Design and Evaluation for Real-Time Multivariate Time Series Fault Detection in Spacecraft Attitude Sensors [41.94295877935867]
  This paper presents a novel approach to detecting stuck values within the Accelerometer and Inertial Measurement Unit of a drone-like spacecraft. A multi-channel Convolutional Neural Network (CNN) is used to perform multi-target classification and independently detect faults in the sensors. An integration methodology is proposed to enable the network to effectively detect anomalies and trigger recovery actions at the system level.
  arXiv  Detail & Related papers  (2024-10-11T09:36:38Z)
• Analyzing Adversarial Inputs in Deep Reinforcement Learning [53.3760591018817]
  We present a comprehensive analysis of the characterization of adversarial inputs, through the lens of formal verification. We introduce a novel metric, the Adversarial Rate, to classify models based on their susceptibility to such perturbations. Our analysis empirically demonstrates how adversarial inputs can affect the safety of a given DRL system with respect to such perturbations.
  arXiv  Detail & Related papers  (2024-02-07T21:58:40Z)
• Effective Intrusion Detection in Heterogeneous Internet-of-Things Networks via Ensemble Knowledge Distillation-based Federated Learning [52.6706505729803]
  We introduce Federated Learning (FL) to collaboratively train a decentralized shared model of Intrusion Detection Systems (IDS) FLEKD enables a more flexible aggregation method than conventional model fusion techniques. Experiment results show that the proposed approach outperforms local training and traditional FL in terms of both speed and performance.
  arXiv  Detail & Related papers  (2024-01-22T14:16:37Z)
• AIOps-Driven Enhancement of Log Anomaly Detection in Unsupervised Scenarios [0.18641315013048293]
  This study introduces a novel hybrid framework through an innovative algorithm that incorporates an unsupervised strategy. The proposed approach encompasses the utilization of both simulated and real-world datasets. The experimental results are highly promising, demonstrating significant reductions in pseudo-positives.
  arXiv  Detail & Related papers  (2023-11-05T11:16:24Z)
• PREM: A Simple Yet Effective Approach for Node-Level Graph Anomaly Detection [65.24854366973794]
  Node-level graph anomaly detection (GAD) plays a critical role in identifying anomalous nodes from graph-structured data in domains such as medicine, social networks, and e-commerce. We introduce a simple method termed PREprocessing and Matching (PREM for short) to improve the efficiency of GAD. Our approach streamlines GAD, reducing time and memory consumption while maintaining powerful anomaly detection capabilities.
  arXiv  Detail & Related papers  (2023-10-18T02:59:57Z)
• Small Object Detection via Coarse-to-fine Proposal Generation and Imitation Learning [52.06176253457522]
  We propose a two-stage framework tailored for small object detection based on the Coarse-to-fine pipeline and Feature Imitation learning. CFINet achieves state-of-the-art performance on the large-scale small object detection benchmarks, SODA-D and SODA-A.
  arXiv  Detail & Related papers  (2023-08-18T13:13:09Z)
• PULL: Reactive Log Anomaly Detection Based On Iterative PU Learning [58.85063149619348]
  We propose PULL, an iterative log analysis method for reactive anomaly detection based on estimated failure time windows. Our evaluation shows that PULL consistently outperforms ten benchmark baselines across three different datasets.
  arXiv  Detail & Related papers  (2023-01-25T16:34:43Z)
• Self-Supervised Training with Autoencoders for Visual Anomaly Detection [61.62861063776813]
  We focus on a specific use case in anomaly detection where the distribution of normal samples is supported by a lower-dimensional manifold. We adapt a self-supervised learning regime that exploits discriminative information during training but focuses on the submanifold of normal examples. We achieve a new state-of-the-art result on the MVTec AD dataset -- a challenging benchmark for visual anomaly detection in the manufacturing domain.
  arXiv  Detail & Related papers  (2022-06-23T14:16:30Z)
• Bayesian Optimization with Machine Learning Algorithms Towards Anomaly Detection [66.05992706105224]
  In this paper, an effective anomaly detection framework is proposed utilizing Bayesian Optimization technique. The performance of the considered algorithms is evaluated using the ISCX 2012 dataset. Experimental results show the effectiveness of the proposed framework in term of accuracy rate, precision, low-false alarm rate, and recall.
  arXiv  Detail & Related papers  (2020-08-05T19:29:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.