Develop End-to-End Anomaly Detection System

• URL: http://arxiv.org/abs/2402.10085v1
• Date: Thu, 1 Feb 2024 09:02:44 GMT
• Title: Develop End-to-End Anomaly Detection System
• Authors: Emanuele Mengoli, Zhiyuan Yao, Wutao Wei
• Abstract summary: Anomaly detection plays a crucial role in ensuring network robustness. We propose an end-to-end anomaly detection model development pipeline. We demonstrate the efficacy of the framework by way of introducing and bench-marking a new forecasting model.
• Score: 3.130722489512822
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Anomaly detection plays a crucial role in ensuring network robustness. However, implementing intelligent alerting systems becomes a challenge when considering scenarios in which anomalies can be caused by both malicious and non-malicious events, leading to the difficulty of determining anomaly patterns. The lack of labeled data in the computer networking domain further exacerbates this issue, impeding the development of robust models capable of handling real-world scenarios. To address this challenge, in this paper, we propose an end-to-end anomaly detection model development pipeline. This framework makes it possible to consume user feedback and enable continuous user-centric model performance evaluation and optimization. We demonstrate the efficacy of the framework by way of introducing and bench-marking a new forecasting model -- named \emph{Lachesis} -- on a real-world networking problem. Experiments have demonstrated the robustness and effectiveness of the two proposed versions of \emph{Lachesis} compared with other models proposed in the literature. Our findings underscore the potential for improving the performance of data-driven products over their life cycles through a harmonized integration of user feedback and iterative development.

Related papers

• Unsupervised Model Diagnosis [49.36194740479798]
  This paper proposes Unsupervised Model Diagnosis (UMO) to produce semantic counterfactual explanations without any user guidance. Our approach identifies and visualizes changes in semantics, and then matches these changes to attributes from wide-ranging text sources.
  arXiv  Detail & Related papers  (2024-10-08T17:59:03Z)
• Towards the generation of hierarchical attack models from cybersecurity vulnerabilities using language models [3.7548609506798494]
  This paper investigates the use of a pre-trained language model and siamese network to discern sibling relationships between text-based cybersecurity vulnerability data.
  arXiv  Detail & Related papers  (2024-10-07T13:05:33Z)
• Explanatory Model Monitoring to Understand the Effects of Feature Shifts on Performance [61.06245197347139]
  We propose a novel approach to explain the behavior of a black-box model under feature shifts. We refer to our method that combines concepts from Optimal Transport and Shapley Values as Explanatory Performance Estimation.
  arXiv  Detail & Related papers  (2024-08-24T18:28:19Z)
• A Reliable Framework for Human-in-the-Loop Anomaly Detection in Time Series [17.08674819906415]
  We introduce HILAD, a novel framework designed to foster a dynamic and bidirectional collaboration between humans and AI. Through our visual interface, HILAD empowers domain experts to detect, interpret, and correct unexpected model behaviors at scale.
  arXiv  Detail & Related papers  (2024-05-06T07:44:07Z)
• GADY: Unsupervised Anomaly Detection on Dynamic Graphs [18.1896489628884]
  We propose a continuous dynamic graph model to capture the fine-grained information, which breaks the limit of existing discrete methods. For the second challenge, we pioneer the use of Generative Adversarial Networks to generate negative interactions. Our proposed GADY significantly outperforms the previous state-of-the-art method on three real-world datasets.
  arXiv  Detail & Related papers  (2023-10-25T05:27:45Z)
• Comprehensive Analysis of Network Robustness Evaluation Based on Convolutional Neural Networks with Spatial Pyramid Pooling [4.366824280429597]
  Connectivity robustness, a crucial aspect for understanding, optimizing, and repairing complex networks, has traditionally been evaluated through simulations. We address these challenges by designing a convolutional neural networks (CNN) model with spatial pyramid pooling networks (SPP-net) We show that the proposed CNN model consistently achieves accurate evaluations of both attack curves and robustness values across all removal scenarios.
  arXiv  Detail & Related papers  (2023-08-10T09:54:22Z)
• On the Robustness of Quality Measures for GANs [136.18799984346248]
  This work evaluates the robustness of quality measures of generative models such as Inception Score (IS) and Fr'echet Inception Distance (FID) We show that such metrics can also be manipulated by additive pixel perturbations.
  arXiv  Detail & Related papers  (2022-01-31T06:43:09Z)
• Explainable Adversarial Attacks in Deep Neural Networks Using Activation Profiles [69.9674326582747]
  This paper presents a visual framework to investigate neural network models subjected to adversarial examples. We show how observing these elements can quickly pinpoint exploited areas in a model.
  arXiv  Detail & Related papers  (2021-03-18T13:04:21Z)
• Non-Singular Adversarial Robustness of Neural Networks [58.731070632586594]
  Adrial robustness has become an emerging challenge for neural network owing to its over-sensitivity to small input perturbations. We formalize the notion of non-singular adversarial robustness for neural networks through the lens of joint perturbations to data inputs as well as model weights.
  arXiv  Detail & Related papers  (2021-02-23T20:59:30Z)
• Firearm Detection via Convolutional Neural Networks: Comparing a Semantic Segmentation Model Against End-to-End Solutions [68.8204255655161]
  Threat detection of weapons and aggressive behavior from live video can be used for rapid detection and prevention of potentially deadly incidents. One way for achieving this is through the use of artificial intelligence and, in particular, machine learning for image analysis. We compare a traditional monolithic end-to-end deep learning model and a previously proposed model based on an ensemble of simpler neural networks detecting fire-weapons via semantic segmentation.
  arXiv  Detail & Related papers  (2020-12-17T15:19:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.