On Generalisability of Machine Learning-based Network Intrusion Detection Systems

• URL: http://arxiv.org/abs/2205.04112v1
• Date: Mon, 9 May 2022 08:26:48 GMT
• Title: On Generalisability of Machine Learning-based Network Intrusion Detection Systems
• Authors: Siamak Layeghy, Marius Portmann
• Abstract summary: In this paper, we evaluate seven supervised and unsupervised learning models on four benchmark NIDS datasets. Our investigation indicates that none of the considered models is able to generalise over all studied datasets. Our investigation also indicates that overall, unsupervised learning methods generalise better than supervised learning models in our considered scenarios.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Many of the proposed machine learning (ML) based network intrusion detection systems (NIDSs) achieve near perfect detection performance when evaluated on synthetic benchmark datasets. Though, there is no record of if and how these results generalise to other network scenarios, in particular to real-world networks. In this paper, we investigate the generalisability property of ML-based NIDSs by extensively evaluating seven supervised and unsupervised learning models on four recently published benchmark NIDS datasets. Our investigation indicates that none of the considered models is able to generalise over all studied datasets. Interestingly, our results also indicate that the generalisability has a high degree of asymmetry, i.e., swapping the source and target domains can significantly change the classification performance. Our investigation also indicates that overall, unsupervised learning methods generalise better than supervised learning models in our considered scenarios. Using SHAP values to explain these results indicates that the lack of generalisability is mainly due to the presence of strong correspondence between the values of one or more features and Attack/Benign classes in one dataset-model combination and its absence in other datasets that have different feature distributions.

Related papers

• On the Cross-Dataset Generalization of Machine Learning for Network Intrusion Detection [50.38534263407915]
  Network Intrusion Detection Systems (NIDS) are a fundamental tool in cybersecurity. Their ability to generalize across diverse networks is a critical factor in their effectiveness and a prerequisite for real-world applications. In this study, we conduct a comprehensive analysis on the generalization of machine-learning-based NIDS through an extensive experimentation in a cross-dataset framework.
  arXiv  Detail & Related papers  (2024-02-15T14:39:58Z)
• Towards out-of-distribution generalization in large-scale astronomical surveys: robust networks learn similar representations [3.653721769378018]
  We use Centered Kernel Alignment (CKA), a similarity measure metric of neural network representations, to examine the relationship between representation similarity and performance. We find that when models are robust to a distribution shift, they produce substantially different representations across their layers on OOD data. We discuss the potential application of similarity representation in guiding model design, training strategy, and mitigating the OOD problem by incorporating CKA as an inductive bias during training.
  arXiv  Detail & Related papers  (2023-11-29T19:00:05Z)
• Regularization Through Simultaneous Learning: A Case Study on Plant Classification [0.0]
  This paper introduces Simultaneous Learning, a regularization approach drawing on principles of Transfer Learning and Multi-task Learning. We leverage auxiliary datasets with the target dataset, the UFOP-HVD, to facilitate simultaneous classification guided by a customized loss function. Remarkably, our approach demonstrates superior performance over models without regularization.
  arXiv  Detail & Related papers  (2023-05-22T19:44:57Z)
• Towards Weakly-Supervised Hate Speech Classification Across Datasets [47.101942709219784]
  We show the effectiveness of a state-of-the-art weakly-supervised text classification model in various in-dataset and cross-dataset settings. We also conduct an in-depth quantitative and qualitative analysis of the source of poor generalizability of HS classification models.
  arXiv  Detail & Related papers  (2023-05-04T08:15:40Z)
• GREAT Score: Global Robustness Evaluation of Adversarial Perturbation using Generative Models [60.48306899271866]
  We present a new framework, called GREAT Score, for global robustness evaluation of adversarial perturbation using generative models. We show high correlation and significantly reduced cost of GREAT Score when compared to the attack-based model ranking on RobustBench. GREAT Score can be used for remote auditing of privacy-sensitive black-box models.
  arXiv  Detail & Related papers  (2023-04-19T14:58:27Z)
• Batch-Ensemble Stochastic Neural Networks for Out-of-Distribution Detection [55.028065567756066]
  Out-of-distribution (OOD) detection has recently received much attention from the machine learning community due to its importance in deploying machine learning models in real-world applications. In this paper we propose an uncertainty quantification approach by modelling the distribution of features. We incorporate an efficient ensemble mechanism, namely batch-ensemble, to construct the batch-ensemble neural networks (BE-SNNs) and overcome the feature collapse problem. We show that BE-SNNs yield superior performance on several OOD benchmarks, such as the Two-Moons dataset, the FashionMNIST vs MNIST dataset, FashionM
  arXiv  Detail & Related papers  (2022-06-26T16:00:22Z)
• General Greedy De-bias Learning [163.65789778416172]
  We propose a General Greedy De-bias learning framework (GGD), which greedily trains the biased models and the base model like gradient descent in functional space. GGD can learn a more robust base model under the settings of both task-specific biased models with prior knowledge and self-ensemble biased model without prior knowledge.
  arXiv  Detail & Related papers  (2021-12-20T14:47:32Z)
• An Explainable Machine Learning-based Network Intrusion Detection System for Enabling Generalisability in Securing IoT Networks [0.0]
  Machine Learning (ML)-based network intrusion detection systems bring many benefits for enhancing the security posture of an organisation. Many systems have been designed and developed in the research community, often achieving a perfect detection rate when evaluated using certain datasets. This paper tightens the gap by evaluating the generalisability of a common feature set to different network environments and attack types.
  arXiv  Detail & Related papers  (2021-04-15T00:44:45Z)
• Anomaly Detection on Attributed Networks via Contrastive Self-Supervised Learning [50.24174211654775]
  We present a novel contrastive self-supervised learning framework for anomaly detection on attributed networks. Our framework fully exploits the local information from network data by sampling a novel type of contrastive instance pair. A graph neural network-based contrastive learning model is proposed to learn informative embedding from high-dimensional attributes and local structure.
  arXiv  Detail & Related papers  (2021-02-27T03:17:20Z)
• CDEvalSumm: An Empirical Study of Cross-Dataset Evaluation for Neural Summarization Systems [121.78477833009671]
  We investigate the performance of different summarization models under a cross-dataset setting. A comprehensive study of 11 representative summarization systems on 5 datasets from different domains reveals the effect of model architectures and generation ways.
  arXiv  Detail & Related papers  (2020-10-11T02:19:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.