A Hierarchical Approach to Conditional Random Fields for System Anomaly Detection

• URL: http://arxiv.org/abs/2210.15030v2
• Date: Fri, 28 Oct 2022 05:33:51 GMT
• Title: A Hierarchical Approach to Conditional Random Fields for System Anomaly Detection
• Authors: Srishti Mishra, Tvarita Jain, Dinkar Sitaram
• Abstract summary: Anomaly detection to recognize unusual events in large scale systems is critical in many industries. A hierarchical approach takes advantage of the implicit relationships in complex systems and localized context.
• Score: 0.8164433158925593
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Anomaly detection to recognize unusual events in large scale systems in a time sensitive manner is critical in many industries, eg. bank fraud, enterprise systems, medical alerts, etc. Large-scale systems often grow in size and complexity over time, and anomaly detection algorithms need to adapt to changing structures. A hierarchical approach takes advantage of the implicit relationships in complex systems and localized context. The features in complex systems may vary drastically in data distribution, capturing different aspects from multiple data sources, and when put together provide a more complete view of the system. In this paper, two datasets are considered, the 1st comprising of system metrics from machines running on a cloud service, and the 2nd of application metrics from a large-scale distributed software system with inherent hierarchies and interconnections amongst its system nodes. Comparing algorithms, across the changepoint based PELT algorithm, cognitive learning-based Hierarchical Temporal Memory algorithms, Support Vector Machines and Conditional Random Fields provides a basis for proposing a Hierarchical Global-Local Conditional Random Field approach to accurately capture anomalies in complex systems across various features. Hierarchical algorithms can learn both the intricacies of specific features, and utilize these in a global abstracted representation to detect anomalous patterns robustly across multi-source feature data and distributed systems. A graphical network analysis on complex systems can further fine-tune datasets to mine relationships based on available features, which can benefit hierarchical models. Furthermore, hierarchical solutions can adapt well to changes at a localized level, learning on new data and changing environments when parts of a system are over-hauled, and translate these learnings to a global view of the system over time.

Related papers

• A Hierarchical Framework with Spatio-Temporal Consistency Learning for Emergence Detection in Complex Adaptive Systems [41.055298739292695]
  Emergence, a global property of complex adaptive systems, is prevalent in real-world dynamic systems, e.g., network-level traffic congestions. This paper proposes a hierarchical framework with CAS-temporal consistency to solve these two problems by learning the system representation and agent representations. Our method achieves more detection than traditional methods and deep learning methods on three datasets with well-known yet hard-to-detect emergent behaviors.
  arXiv  Detail & Related papers  (2024-01-18T08:55:05Z)
• Interactive System-wise Anomaly Detection [66.3766756452743]
  Anomaly detection plays a fundamental role in various applications. It is challenging for existing methods to handle the scenarios where the instances are systems whose characteristics are not readily observed as data. We develop an end-to-end approach which includes an encoder-decoder module that learns system embeddings.
  arXiv  Detail & Related papers  (2023-04-21T02:20:24Z)
• Heterogeneous Anomaly Detection for Software Systems via Semi-supervised Cross-modal Attention [29.654681594903114]
  We propose Hades, the first end-to-end semi-supervised approach to identify system anomalies based on heterogeneous data. Our approach employs a hierarchical architecture to learn a global representation of the system status by fusing log semantics and metric patterns. We evaluate Hades extensively on large-scale simulated data and datasets from Huawei Cloud.
  arXiv  Detail & Related papers  (2023-02-14T09:02:11Z)
• A Survey of Graph-based Deep Learning for Anomaly Detection in Distributed Systems [2.3551989288556774]
  We explore the potentials of graph-based algorithms to identify anomalies in distributed systems. One of our objectives is to provide an in-depth look at graph-based approaches to conceptually analyze their capability to handle real-world challenges. This study gives an overview of the State-of-the-Art (SotA) research articles in the field and compare and contrast their characteristics.
  arXiv  Detail & Related papers  (2022-06-08T20:19:28Z)
• CSCAD: Correlation Structure-based Collective Anomaly Detection in Complex System [11.739889613196619]
  We propose a correlation structure-based collective anomaly detection model for high-dimensional anomaly detection problem in large systems. Our framework utilize graph convolutional network combining a variational autoencoder to jointly exploit the feature space correlation and reconstruction deficiency of samples. An anomaly discriminating network can then be trained using low anomalous degree samples as positive samples, and high anomalous degree samples as negative samples.
  arXiv  Detail & Related papers  (2021-05-30T09:28:25Z)
• Clustered Federated Learning via Generalized Total Variation Minimization [83.26141667853057]
  We study optimization methods to train local (or personalized) models for local datasets with a decentralized network structure. Our main conceptual contribution is to formulate federated learning as total variation minimization (GTV) Our main algorithmic contribution is a fully decentralized federated learning algorithm.
  arXiv  Detail & Related papers  (2021-05-26T18:07:19Z)
• Learning Dependencies in Distributed Cloud Applications to Identify and Localize Anomalies [58.88325379746632]
  We present Arvalus and its variant D-Arvalus, a neural graph transformation method that models system components as nodes and their dependencies as edges to improve the identification and localization of anomalies. Given a series of metric, our method predicts the most likely system state - either normal or an anomaly class - and performs localization when an anomaly is detected. The evaluation shows the generally good prediction performance of Arvalus and reveals the advantage of D-Arvalus which incorporates information about system component dependencies.
  arXiv  Detail & Related papers  (2021-03-09T06:34:05Z)
• Anomaly Detection on Attributed Networks via Contrastive Self-Supervised Learning [50.24174211654775]
  We present a novel contrastive self-supervised learning framework for anomaly detection on attributed networks. Our framework fully exploits the local information from network data by sampling a novel type of contrastive instance pair. A graph neural network-based contrastive learning model is proposed to learn informative embedding from high-dimensional attributes and local structure.
  arXiv  Detail & Related papers  (2021-02-27T03:17:20Z)
• TELESTO: A Graph Neural Network Model for Anomaly Classification in Cloud Services [77.454688257702]
  Machine learning (ML) and artificial intelligence (AI) are applied on IT system operation and maintenance. One direction aims at the recognition of re-occurring anomaly types to enable remediation automation. We propose a method that is invariant to dimensionality changes of given data.
  arXiv  Detail & Related papers  (2021-02-25T14:24:49Z)
• Self-organizing Democratized Learning: Towards Large-scale Distributed Learning Systems [71.14339738190202]
  democratized learning (Dem-AI) lays out a holistic philosophy with underlying principles for building large-scale distributed and democratized machine learning systems. Inspired by Dem-AI philosophy, a novel distributed learning approach is proposed in this paper. The proposed algorithms demonstrate better results in the generalization performance of learning models in agents compared to the conventional FL algorithms.
  arXiv  Detail & Related papers  (2020-07-07T08:34:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.