An Evaluation Framework for Network IDS/IPS Datasets: Leveraging MITRE ATT&CK and Industry Relevance Metrics

• URL: http://arxiv.org/abs/2511.12743v1
• Date: Sun, 16 Nov 2025 19:17:00 GMT
• Title: An Evaluation Framework for Network IDS/IPS Datasets: Leveraging MITRE ATT&CK and Industry Relevance Metrics
• Authors: Adrita Rahman Tori, Khondokar Fida Hasan,
• Abstract summary: Current AI model evaluation practices for developing IDS/IPS focus predominantly on accuracy metrics.<n>We introduce a novel multi-dimensional framework that integrates the MITRE ATT&CK knowledge base for threat intelligence.<n>Applying this framework to nine publicly available IDS/IPS datasets reveals significant gaps in threat coverage.
• Score: 1.6006586061577803
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The performance of Machine Learning (ML) and Deep Learning (DL)-based Intrusion Detection and Prevention Systems (IDS/IPS) is critically dependent on the relevance and quality of the datasets used for training and evaluation. However, current AI model evaluation practices for developing IDS/IPS focus predominantly on accuracy metrics, often overlooking whether datasets represent industry-specific threats. To address this gap, we introduce a novel multi-dimensional framework that integrates the MITRE ATT&CK knowledge base for threat intelligence and employs five complementary metrics that together provide a comprehensive assessment of dataset suitability. Methodologically, this framework combines threat intelligence, natural language processing, and quantitative analysis to assess the suitability of datasets for specific industry contexts. Applying this framework to nine publicly available IDS/IPS datasets reveals significant gaps in threat coverage, particularly in the healthcare, energy, and financial sectors. In particular, recent datasets (e.g., CIC-IoMT, CIC-UNSW-NB15) align better with sector-specific threats, whereas others, like CICIoV-24, underperform despite their recency. Our findings provide a standardized, interpretable approach for selecting datasets aligned with sector-specific operational requirements, ultimately enhancing the real-world effectiveness of AI-driven IDS/IPS deployments. The efficiency and practicality of the framework are validated through deployment in a real-world case study, underscoring its capacity to inform dataset selection and enhance the effectiveness of AI-driven IDS/IPS in operational environments.

Related papers

• Decision Quality Evaluation Framework at Pinterest [0.36944296923226316]
  The framework is centered on a high-trust Golden Set (GDS) curated by subject matter experts (SMEs)<n>We introduce an automated intelligent sampling pipeline that uses propensity scores to efficiently expand dataset coverage.<n>The framework enables a shift from subjective assessments to a data-driven and quantitative practice for managing content safety systems.
  arXiv  Detail & Related papers  (2026-02-17T18:45:55Z)
• Multi-Agent Collaborative Intrusion Detection for Low-Altitude Economy IoT: An LLM-Enhanced Agentic AI Framework [60.72591149679355]
  The rapid expansion of low-altitude economy Internet of Things (LAE-IoT) networks has created unprecedented security challenges.<n>Traditional intrusion detection systems fail to tackle the unique characteristics of aerial IoT environments.<n>We introduce a large language model (LLM)-enabled agentic AI framework for enhancing intrusion detection in LAE-IoT networks.
  arXiv  Detail & Related papers  (2026-01-25T12:47:25Z)
• A Framework for Data Valuation and Monetisation [0.0]
  This paper introduces a unified valuation framework that integrates economic, governance, and strategic perspectives into a coherent decision-support model.<n>The model combines qualitative scoring, cost- and utility-based estimation, relevance/quality indexing, and multi-criteria weighting to define data value transparently and systematically.
  arXiv  Detail & Related papers  (2025-12-08T15:57:26Z)
• Financial Data Analysis with Robust Federated Logistic Regression [7.68275287892947]
  In this study, we focus on the analysis of financial data in a federated setting, wherein data is distributed across multiple clients or locations.<n>We propose a robust federated logistic regression-based framework that strives to strike a balance between these goals.
  arXiv  Detail & Related papers  (2025-04-28T20:42:24Z)
• VirtualXAI: A User-Centric Framework for Explainability Assessment Leveraging GPT-Generated Personas [0.07499722271664146]
  The demand for eXplainable AI (XAI) has increased to enhance the interpretability, transparency, and trustworthiness of AI models.<n>We propose a framework that integrates quantitative benchmarking with qualitative user assessments through virtual personas.<n>This yields an estimated XAI score and provides tailored recommendations for both the optimal AI model and the XAI method for a given scenario.
  arXiv  Detail & Related papers  (2025-03-06T09:44:18Z)
• Larger or Smaller Reward Margins to Select Preferences for Alignment? [47.11487070429289]
  Preference learning is critical for aligning large language models with human values.<n>We introduce the alignment potential metric, which quantifies the gap from the model's current implicit reward margin to the target explicit reward margin.<n> Empirical results demonstrate that training on data selected by this metric consistently enhances alignment performance.
  arXiv  Detail & Related papers  (2025-02-25T06:43:24Z)
• Outside the Comfort Zone: Analysing LLM Capabilities in Software Vulnerability Detection [9.652886240532741]
  This paper thoroughly analyses large language models' capabilities in detecting vulnerabilities within source code. We evaluate the performance of six open-source models that are specifically trained for vulnerability detection against six general-purpose LLMs.
  arXiv  Detail & Related papers  (2024-08-29T10:00:57Z)
• Footprints of Data in a Classifier: Understanding the Privacy Risks and Solution Strategies [0.9208007322096533]
  Article 17 of the General Data Protection Regulation (Right Erasure) requires data to be permanently removed from a system to prevent potential compromise.<n>One such issue arises from the residual footprints of training data embedded within predictive models.<n>This study examines how two fundamental aspects of classifier systems - training quality and classifier training methodology - contribute to privacy vulnerabilities.
  arXiv  Detail & Related papers  (2024-07-02T13:56:37Z)
• Enhancing IoT Security Against DDoS Attacks through Federated Learning [0.0]
  Internet of Things (IoT) has ushered in transformative connectivity between physical devices and the digital realm. Traditional DDoS mitigation approaches are ill-equipped to handle the intricacies of IoT ecosystems. This paper introduces an innovative strategy to bolster the security of IoT networks against DDoS attacks by harnessing the power of Federated Learning.
  arXiv  Detail & Related papers  (2024-03-16T16:45:28Z)
• Data Poisoning for In-context Learning [49.77204165250528]
  In-context learning (ICL) has been recognized for its innovative ability to adapt to new tasks.<n>This paper delves into the critical issue of ICL's susceptibility to data poisoning attacks.<n>We introduce ICLPoison, a specialized attacking framework conceived to exploit the learning mechanisms of ICL.
  arXiv  Detail & Related papers  (2024-02-03T14:20:20Z)
• When is Off-Policy Evaluation (Reward Modeling) Useful in Contextual Bandits? A Data-Centric Perspective [64.73162159837956]
  evaluating the value of a hypothetical target policy with only a logged dataset is important but challenging. We propose DataCOPE, a data-centric framework for evaluating a target policy given a dataset. Our empirical analysis of DataCOPE in the logged contextual bandit settings using healthcare datasets confirms its ability to evaluate both machine-learning and human expert policies.
  arXiv  Detail & Related papers  (2023-11-23T17:13:37Z)
• Uncertainty Estimation by Fisher Information-based Evidential Deep Learning [61.94125052118442]
  Uncertainty estimation is a key factor that makes deep learning reliable in practical applications. We propose a novel method, Fisher Information-based Evidential Deep Learning ($mathcalI$-EDL) In particular, we introduce Fisher Information Matrix (FIM) to measure the informativeness of evidence carried by each sample, according to which we can dynamically reweight the objective loss terms to make the network more focused on the representation learning of uncertain classes.
  arXiv  Detail & Related papers  (2023-03-03T16:12:59Z)
• Distributed intelligence on the Edge-to-Cloud Continuum: A systematic literature review [62.997667081978825]
  This review aims at providing a comprehensive vision of the main state-of-the-art libraries and frameworks for machine learning and data analytics available today. The main simulation, emulation, deployment systems, and testbeds for experimental research on the Edge-to-Cloud Continuum available today are also surveyed.
  arXiv  Detail & Related papers  (2022-04-29T08:06:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.