How Benchmarks and Evaluation Protocols Shape Conclusions in Provenance-Based Intrusion Detection

• URL: http://arxiv.org/abs/2608.01454v1
• Date: Sun, 02 Aug 2026 19:27:49 GMT
• Title: How Benchmarks and Evaluation Protocols Shape Conclusions in Provenance-Based Intrusion Detection
• Abstract summary: We re-evaluate representative PIDS on public datasets that meet our audit, labeling, and calibration requirements.<n>We find that alerting success and investigation utility can diverge sharply, as several systems surface attacks without providing enough process-level context to support forensic investigation.<n>This analysis helps explain why several audited E3 datasets can expose alerting behavior without reliably separating model architectures.
• Score: 6.453778601809096
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Provenance-based intrusion detection systems (PIDS) frequently report strong performance, but the conclusions drawn from these results can be highly sensitive to benchmarking choices and evaluation protocols. We investigate this dependency by re-evaluating representative PIDS on public datasets that meet our audit, labeling, and calibration requirements. Focusing primarily on the audited DARPA TC E3 datasets, we apply a unified protocol with temporally separated test periods and validation-only checkpoint and threshold calibration, and ask which architectural claims are empirically supported. We find that alerting success and investigation utility can diverge sharply, as several systems surface attacks without providing enough process-level context to support forensic investigation. On three of the four primary datasets, a simple allowlist built from training executable names and paths matches or exceeds the selected learned baselines on key operating-point metrics, suggesting that much of their measured performance reflects lexical novelty rather than richer provenance modeling. To explain why only some datasets expose architectural differences, we measure semantic signal quality through feature completeness and field entropy. This analysis helps explain why several audited E3 datasets can expose alerting behavior without reliably separating model architectures, while Theia pairs the strongest semantic signal quality with the clearest improvements in ranking and node-level recovery by our reference model. These results show that architectural claims in PIDS should be interpreted together with the benchmark properties and evaluation protocol that produced them.

Related papers

• Mind the Gap: Robustness Risks in PII Detection Systems [6.167318681955322]
  We study how different PII detection paradigms fail under noisy, unstructured, and informal inputs.<n>No single architecture is uniformly reliable across PII categories.<n>Motivated by these findings, we propose a hybrid detection pipeline with a QA-driven feedback loop for iterative risk mitigation.
  arXiv  Detail & Related papers  (2026-09-03T07:23:28Z)
• MedSegBenchmarker: A Raw-Count-First Framework for Controlled 2D Medical Image Segmentation Benchmarks [1.0230975056328007]
  We present MEDSEGBENCHMARKER (MSB), a configuration-driven framework for controlled benchmarking of 2D datasets.<n>MSB exports sample- and class-level pixel counts and predictions together with the evaluation context.<n>We demonstrate MSB in a case study involving three heterogeneous 2D datasets and multiple vision models evaluated at 256- and 512-pixel input resolutions.
  arXiv  Detail & Related papers  (2026-08-30T09:20:38Z)
• From Benchmark Performance to Tool Deployment: Human-in-the-Loop Anomaly Detection [28.67239775192971]
  We evaluate 19 unsupervised anomaly detection models on the BowTie dataset.<n>We observe that model performance is less stable than typically reported on standard benchmarks.<n>We deploy a unified human-in-the-loop framework for manufactured-part inspection.
  arXiv  Detail & Related papers  (2026-08-07T21:31:20Z)
• From Signals to Transfer: A Factorised Study of Probe-Based Uncertainty Estimation in Large Language Models [51.470637128942336]
  We propose a factorised study of probe-based uncertainty estimation (UE) under matched conditions.<n>Our results show that raw hidden states and attention features are difficult to outperform in-domain.<n>We train benchmark-based pretrained probes that transfer reasonably well to open-ended factual generation.
  arXiv  Detail & Related papers  (2026-06-26T03:23:15Z)
• Unsupervised Skill Discovery for Agentic Data Analysis [55.42853694128929]
  DataCOPE is an unsupervised verifier-guided skill discovery framework for data-analytic agents.<n>It iteratively coordinates a Data-Analytic Agent for trajectory generation, an Unsupervised Verifier for signal extraction, and a Skill Manager for contrastive skill distillation.<n>We evaluate DataCOPE on report-style analysis from Deep Data Research and reasoning-style analysis from DABStep.
  arXiv  Detail & Related papers  (2026-06-04T17:20:47Z)
• NeuroTrace: Inference Provenance-Based Detection of Adversarial Examples [1.096626056612224]
  We introduce NeuroTrace, a framework for analyzing inference provenance through Inference Provenance Graphs (IPGs)<n>IPGs are heterogeneous graphs that capture both activation behavior and parameter-induced dataflow during a model's forward pass.<n>We evaluate IPG-based detectors for adversarial example detection under intra-attack, multi-attack, and cross-threat transfer settings.
  arXiv  Detail & Related papers  (2026-04-15T22:23:40Z)
• Heuristic-inspired Reasoning Priors Facilitate Data-Efficient Referring Object Detection [53.988759250627425]
  HeROD (Heuristic-inspired ROD) is a lightweight, model-agnostic framework that injects explicit, interpretable spatial and semantic reasonings.<n>HeROD consistently outperforms strong grounding baselines in scarcelabel regimes.
  arXiv  Detail & Related papers  (2026-03-25T10:33:22Z)
• PIDSMaker: Building and Evaluating Provenance-based Intrusion Detection Systems [0.5830619388189558]
  PIDSMaker is an open-source framework for developing and evaluating PIDSs under consistent visualization protocols.<n>It consolidates eight state-of-the-art systems into a modular architecture with standardized preprocessing and ground-truth labels.<n>A YAML-based configuration interface supports rapid prototyping by composing components across systems without code changes.
  arXiv  Detail & Related papers  (2026-01-30T13:45:33Z)
• Demystifying Network Foundation Models [14.775836458250799]
  This work presents a systematic investigation into the latent knowledge encoded within Network Foundation Models (NFMs)<n>We evaluate four state-of-the-art NFMs, revealing that they all exhibit significant anisotropy, inconsistent feature sensitivity patterns.<n>Our work identifies numerous limitations across all models and demonstrates that addressing them can significantly improve model performance.
  arXiv  Detail & Related papers  (2025-09-27T03:53:46Z)
• DeepfakeBench: A Comprehensive Benchmark of Deepfake Detection [55.70982767084996]
  A critical yet frequently overlooked challenge in the field of deepfake detection is the lack of a standardized, unified, comprehensive benchmark. We present the first comprehensive benchmark for deepfake detection, called DeepfakeBench, which offers three key contributions. DeepfakeBench contains 15 state-of-the-art detection methods, 9CL datasets, a series of deepfake detection evaluation protocols and analysis tools, as well as comprehensive evaluations.
  arXiv  Detail & Related papers  (2023-07-04T01:34:41Z)
• Revisiting the Evaluation of Image Synthesis with GANs [55.72247435112475]
  This study presents an empirical investigation into the evaluation of synthesis performance, with generative adversarial networks (GANs) as a representative of generative models. In particular, we make in-depth analyses of various factors, including how to represent a data point in the representation space, how to calculate a fair distance using selected samples, and how many instances to use from each set.
  arXiv  Detail & Related papers  (2023-04-04T17:54:32Z)
• WSSOD: A New Pipeline for Weakly- and Semi-Supervised Object Detection [75.80075054706079]
  We propose a weakly- and semi-supervised object detection framework (WSSOD) An agent detector is first trained on a joint dataset and then used to predict pseudo bounding boxes on weakly-annotated images. The proposed framework demonstrates remarkable performance on PASCAL-VOC and MSCOCO benchmark, achieving a high performance comparable to those obtained in fully-supervised settings.
  arXiv  Detail & Related papers  (2021-05-21T11:58:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.