SoK: Pragmatic Assessment of Machine Learning for Network Intrusion Detection

• URL: http://arxiv.org/abs/2305.00550v1
• Date: Sun, 30 Apr 2023 18:38:27 GMT
• Title: SoK: Pragmatic Assessment of Machine Learning for Network Intrusion Detection
• Authors: Giovanni Apruzzese, Pavel Laskov, Johannes Schneider
• Abstract summary: This paper aims to reduce the practitioners' skepticism towards ML for NID by "changing" the evaluation methodology adopted in research. We propose the notion of "pragmatic assessment", which enable practitioners to gauge the real value of ML methods for NID.
• Score: 3.8759572154478343
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Machine Learning (ML) has become a valuable asset to solve many real-world tasks. For Network Intrusion Detection (NID), however, scientific advances in ML are still seen with skepticism by practitioners. This disconnection is due to the intrinsically limited scope of research papers, many of which primarily aim to demonstrate new methods ``outperforming'' prior work -- oftentimes overlooking the practical implications for deploying the proposed solutions in real systems. Unfortunately, the value of ML for NID depends on a plethora of factors, such as hardware, that are often neglected in scientific literature. This paper aims to reduce the practitioners' skepticism towards ML for NID by "changing" the evaluation methodology adopted in research. After elucidating which "factors" influence the operational deployment of ML in NID, we propose the notion of "pragmatic assessment", which enable practitioners to gauge the real value of ML methods for NID. Then, we show that the state-of-research hardly allows one to estimate the value of ML for NID. As a constructive step forward, we carry out a pragmatic assessment. We re-assess existing ML methods for NID, focusing on the classification of malicious network traffic, and consider: hundreds of configuration settings; diverse adversarial scenarios; and four hardware platforms. Our large and reproducible evaluations enable estimating the quality of ML for NID. We also validate our claims through a user-study with security practitioners.

Related papers

• MR-Ben: A Meta-Reasoning Benchmark for Evaluating System-2 Thinking in LLMs [55.20845457594977]
  Large language models (LLMs) have shown increasing capability in problem-solving and decision-making. We present a process-based benchmark MR-Ben that demands a meta-reasoning skill. Our meta-reasoning paradigm is especially suited for system-2 slow thinking.
  arXiv  Detail & Related papers  (2024-06-20T03:50:23Z)
• Common pitfalls to avoid while using multiobjective optimization in machine learning [1.2499537119440245]
  There has been an increasing interest in exploring the application of multiobjective optimization (MOO) in machine learning (ML) Despite its potential, there is a noticeable lack of satisfactory literature that could serve as an entry-level guide for ML practitioners who want to use MOO. We critically review previous studies, particularly those involving MOO in deep learning (using Physics-Informed Neural Networks (PINNs) as a guiding example) and identify misconceptions that highlight the need for a better grasp of MOO principles in ML.
  arXiv  Detail & Related papers  (2024-05-02T17:12:25Z)
• MLLM-Bench: Evaluating Multimodal LLMs with Per-sample Criteria [49.500322937449326]
  Multimodal large language models (MLLMs) have broadened the scope of AI applications. Existing automatic evaluation methodologies for MLLMs are mainly limited in evaluating queries without considering user experiences. We propose a new evaluation paradigm for MLLMs, which is evaluating MLLMs with per-sample criteria using potent MLLM as the judge.
  arXiv  Detail & Related papers  (2023-11-23T12:04:25Z)
• Performance evaluation of Machine learning algorithms for Intrusion Detection System [0.40964539027092917]
  This paper focuses on intrusion detection systems (IDSs) analysis using Machine Learning (ML) techniques. We analyze the KDD CUP-'99' intrusion detection dataset used for training and validating ML models.
  arXiv  Detail & Related papers  (2023-10-01T06:35:37Z)
• A Generative Framework for Low-Cost Result Validation of Machine Learning-as-a-Service Inference [4.478182379059458]
  Fides is a novel framework for real-time integrity validation of ML-as-a-Service (ML) inference. Fides features a client-side attack detection model that uses statistical analysis and divergence measurements to identify, with a high likelihood, if the service model is under attack. We devised a generative adversarial network framework for training the attack detection and re-classification models.
  arXiv  Detail & Related papers  (2023-03-31T19:17:30Z)
• Uncertainty Estimation by Fisher Information-based Evidential Deep Learning [61.94125052118442]
  Uncertainty estimation is a key factor that makes deep learning reliable in practical applications. We propose a novel method, Fisher Information-based Evidential Deep Learning ($mathcalI$-EDL) In particular, we introduce Fisher Information Matrix (FIM) to measure the informativeness of evidence carried by each sample, according to which we can dynamically reweight the objective loss terms to make the network more focused on the representation learning of uncertain classes.
  arXiv  Detail & Related papers  (2023-03-03T16:12:59Z)
• Absolute Wrong Makes Better: Boosting Weakly Supervised Object Detection via Negative Deterministic Information [54.35679298764169]
  Weakly supervised object detection (WSOD) is a challenging task, in which image-level labels are used to train an object detector. This paper focuses on identifying and fully exploiting the deterministic information in WSOD. We propose a negative deterministic information (NDI) based method for improving WSOD, namely NDI-WSOD.
  arXiv  Detail & Related papers  (2022-04-21T12:55:27Z)
• On the Practicality of Deterministic Epistemic Uncertainty [106.06571981780591]
  deterministic uncertainty methods (DUMs) achieve strong performance on detecting out-of-distribution data. It remains unclear whether DUMs are well calibrated and can seamlessly scale to real-world applications.
  arXiv  Detail & Related papers  (2021-07-01T17:59:07Z)
• Active Learning for Network Traffic Classification: A Technical Survey [1.942265343737899]
  This study investigates the applicability of an active form of ML, called Active Learning (AL), which reduces the need for a high number of labeled examples. The study first provides an overview of NTC and its fundamental challenges along with surveying the literature in the field of using ML techniques in NTC. Further, challenges and open issues in the use of AL for NTC are discussed.
  arXiv  Detail & Related papers  (2021-06-13T06:37:50Z)
• Practical Machine Learning Safety: A Survey and Primer [81.73857913779534]
  Open-world deployment of Machine Learning algorithms in safety-critical applications such as autonomous vehicles needs to address a variety of ML vulnerabilities. New models and training techniques to reduce generalization error, achieve domain adaptation, and detect outlier examples and adversarial attacks. Our organization maps state-of-the-art ML techniques to safety strategies in order to enhance the dependability of the ML algorithm from different aspects.
  arXiv  Detail & Related papers  (2021-06-09T05:56:42Z)
• Insights into Performance Fitness and Error Metrics for Machine Learning [1.827510863075184]
  Machine learning (ML) is the field of training machines to achieve high level of cognition and perform human-like analysis. This paper examines a number of the most commonly-used performance fitness and error metrics for regression and classification algorithms.
  arXiv  Detail & Related papers  (2020-05-17T22:59:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.