How Far Should We Need to Go : Evaluate Provenance-based Intrusion Detection Systems in Industrial Scenarios

• URL: http://arxiv.org/abs/2603.22982v1
• Date: Tue, 24 Mar 2026 09:20:33 GMT
• Title: How Far Should We Need to Go : Evaluate Provenance-based Intrusion Detection Systems in Industrial Scenarios
• Abstract summary: We conduct the first systematic evaluation and analysis of PIDSes in industrial scenarios.<n>We identify three main new characteristics in industry: heterogeneous multi-source inputs, more powerful attackers, and increasing benign activity complexity.<n>The evaluation results reveal challenges for existing PIDSes, including poor portability across different hosts and platforms, low detection performance against real-world attacks, and high false positive rates with ever-changing benign activities.
• Score: 20.56094280821142
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Provenance-based Intrusion Detection Systems (PIDSes) have been widely used to detect Advanced Persistent Threats (APTs). Although many studies achieve high performance in the evaluations of their original papers, their performance in industrial scenarios remains unclear. To fill this gap, we conduct the first systematic evaluation and analysis of PIDSes in industrial scenarios. We first analyze the differences between the data from DARPA datasets and that collected in industrial scenarios, identifying three main new characteristics in industry: heterogeneous multi-source inputs, more powerful attackers, and increasing benign activity complexity. We then build several datasets to evaluate five state-of-the-art PIDSes. The evaluation results reveal challenges for existing PIDSes, including poor portability across different hosts and platforms, low detection performance against real-world attacks, and high false positive rates with ever-changing benign activities. Based on the evaluation results and our industrial practices, we provide several insights to solve or explain the above problems. For example, we propose a method to mitigate the high false positives, which reduces manual effort by 2/3. Finally, we propose several research suggestions to improve PIDSes.

Related papers

• How Benchmarks and Evaluation Protocols Shape Conclusions in Provenance-Based Intrusion Detection [6.453778601809096]
  We re-evaluate representative PIDS on public datasets that meet our audit, labeling, and calibration requirements.<n>We find that alerting success and investigation utility can diverge sharply, as several systems surface attacks without providing enough process-level context to support forensic investigation.<n>This analysis helps explain why several audited E3 datasets can expose alerting behavior without reliably separating model architectures.
  arXiv  Detail & Related papers  (2026-08-02T19:27:49Z)
• HIDBench: Benchmarking Large Language Models for Host-Based Intrusion Detection [11.060899899257755]
  We present a new benchmark to assess large language models' capabilities in supporting host-based intrusion detection systems.<n>This task requires fine-grained reasoning over large-scale, noisy, and highly imbalanced system logs.<n>Our results highlight that while LLMs show strong potential for HIDS, their effectiveness is highly sensitive to data complexity.
  arXiv  Detail & Related papers  (2026-05-20T22:07:12Z)
• Contamination Detection for VLMs using Multi-Modal Semantic Perturbation [73.76465227729818]
  Open-source Vision-Language Models (VLMs) have achieved state-of-the-art performance on benchmark tasks.<n>Pretraining corpora raise a critical concern for both practitioners and users: inflated performance due to test-set leakage.<n>We show that existing detection approaches either fail outright or exhibit inconsistent behavior.<n>We propose a novel simple yet effective detection method based on multi-modal semantic perturbation.
  arXiv  Detail & Related papers  (2025-11-05T18:59:52Z)
• Data-Efficient Adaptation and a Novel Evaluation Method for Aspect-based Sentiment Analysis [2.3949433621799283]
  ABSA research needs are unmet in high-demand yet low-resource areas such as education and healthcare.<n>Traditional evaluation methods based on exact matches are overly rigid for ABSA tasks.<n>We propose a novel evaluation method, Flexible Text Similarity Matching and Optimal Bipartite Pairing.<n>We present the first ABSA study of small decoder-only generative language models.
  arXiv  Detail & Related papers  (2025-11-04T22:11:10Z)
• Demystifying deep search: a holistic evaluation with hint-free multi-hop questions and factorised metrics [89.1999907891494]
  We present WebDetective, a benchmark of hint-free multi-hop questions paired with a controlled Wikipedia sandbox.<n>Our evaluation of 25 state-of-the-art models reveals systematic weaknesses across all architectures.<n>We develop an agentic workflow, EvidenceLoop, that explicitly targets the challenges our benchmark identifies.
  arXiv  Detail & Related papers  (2025-10-01T07:59:03Z)
• DEFENDCLI: {Command-Line} Driven Attack Provenance Examination [18.873697430574573]
  DEFENDCLI is an innovative system that delves into command-line-level detection.<n>It improves the precision of the information representation by evaluating infrequent process calls, suspicious command-line executions, and external network connections.<n>Our evaluation demonstrates that DEFENDCLI improves precision by approximately 1.6x compared to the state-of-the-art methods.
  arXiv  Detail & Related papers  (2025-08-18T01:13:27Z)
• Dissecting Long-Chain-of-Thought Reasoning Models: An Empirical Study [91.78803511141975]
  This work focuses on the roles of positive and negative samples in scaling reinforcement learning.<n>We identify substantial data inefficiency in group relative policy optimization, where over half of the samples yield zero advantage.<n>We investigate unstable performance across various reasoning models and benchmarks, attributing instability to uncertain problems with ambiguous outcomes.
  arXiv  Detail & Related papers  (2025-06-05T11:47:10Z)
• Out-of-Distribution Detection: A Task-Oriented Survey of Recent Advances [40.27656150526273]
  Out-of-distribution (OOD) detection aims to detect test samples outside the training category space.<n>We provide a discussion of the evaluation scenarios, a variety of applications, and several future research directions.
  arXiv  Detail & Related papers  (2024-09-18T11:30:30Z)
• Dissecting Out-of-Distribution Detection and Open-Set Recognition: A Critical Analysis of Methods and Benchmarks [17.520137576423593]
  We aim to provide a consolidated view of the two largest sub-fields within the community: out-of-distribution (OOD) detection and open-set recognition (OSR) We perform rigorous cross-evaluation between state-of-the-art methods in the OOD detection and OSR settings and identify a strong correlation between the performances of methods for them. We propose a new, large-scale benchmark setting which we suggest better disentangles the problem tackled by OOD detection and OSR.
  arXiv  Detail & Related papers  (2024-08-29T17:55:07Z)
• A Comprehensive Library for Benchmarking Multi-class Visual Anomaly Detection [89.92916473403108]
  This paper proposes a comprehensive visual anomaly detection benchmark, ADer, which is a modular framework for new methods.<n>The benchmark includes multiple datasets from industrial and medical domains, implementing fifteen state-of-the-art methods and nine comprehensive metrics.<n>We objectively reveal the strengths and weaknesses of different methods and provide insights into the challenges and future directions of multi-class visual anomaly detection.
  arXiv  Detail & Related papers  (2024-06-05T13:40:07Z)
• DeepfakeBench: A Comprehensive Benchmark of Deepfake Detection [55.70982767084996]
  A critical yet frequently overlooked challenge in the field of deepfake detection is the lack of a standardized, unified, comprehensive benchmark. We present the first comprehensive benchmark for deepfake detection, called DeepfakeBench, which offers three key contributions. DeepfakeBench contains 15 state-of-the-art detection methods, 9CL datasets, a series of deepfake detection evaluation protocols and analysis tools, as well as comprehensive evaluations.
  arXiv  Detail & Related papers  (2023-07-04T01:34:41Z)
• Uncertainty-Aware Instance Reweighting for Off-Policy Learning [63.31923483172859]
  We propose a Uncertainty-aware Inverse Propensity Score estimator (UIPS) for improved off-policy learning. Experiment results on synthetic and three real-world recommendation datasets demonstrate the advantageous sample efficiency of the proposed UIPS estimator.
  arXiv  Detail & Related papers  (2023-03-11T11:42:26Z)
• Benchmarking Deep Models for Salient Object Detection [67.07247772280212]
  We construct a general SALient Object Detection (SALOD) benchmark to conduct a comprehensive comparison among several representative SOD methods. In the above experiments, we find that existing loss functions usually specialized in some metrics but reported inferior results on the others. We propose a novel Edge-Aware (EA) loss that promotes deep networks to learn more discriminative features by integrating both pixel- and image-level supervision signals.
  arXiv  Detail & Related papers  (2022-02-07T03:43:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.