StealthCup: Realistic, Multi-Stage, Evasion-Focused CTF for Benchmarking IDS

• URL: http://arxiv.org/abs/2511.17761v1
• Date: Fri, 21 Nov 2025 20:17:59 GMT
• Title: StealthCup: Realistic, Multi-Stage, Evasion-Focused CTF for Benchmarking IDS
• Abstract summary: Intrusion Detection Systems (IDS) are critical to defending enterprise and industrial control environments.<n>Existing benchmarks rely on synthetic datasets which fail to capture adaptive adversary behavior.<n>We present StealthCup, a novel evaluation methodology that operationalizes IDS benchmarking as an evasion-focused Capture-the-Flag competition.
• Score: 1.9715907660202943
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Intrusion Detection Systems (IDS) are critical to defending enterprise and industrial control environments, yet evaluating their effectiveness under realistic conditions remains an open challenge. Existing benchmarks rely on synthetic datasets (e.g., NSL-KDD, CICIDS2017) or scripted replay frameworks, which fail to capture adaptive adversary behavior. Even MITRE ATT&CK Evaluations, while influential, are host-centric and assume malware-driven compromise, thereby under-representing stealthy, multi-stage intrusions across IT and OT domains. We present StealthCup, a novel evaluation methodology that operationalizes IDS benchmarking as an evasion-focused Capture-the-Flag competition. Professional penetration testers engaged in multi-stage attack chains on a realistic IT/OT testbed, with scoring penalizing IDS detections. The event generated structured attacker writeups, validated detections, and PCAPs, host logs, and alerts. Our results reveal that out of 32 exercised attack techniques, 11 were not detected by any IDS configuration. Open-source systems (Wazuh, Suricata) produced high false-positive rates >90%, while commercial tools generated fewer false positives but also missed more attacks. Comparison with the Volt Typhoon APT advisory confirmed strong realism: all 28 applicable techniques were exercised, 19 appeared in writeups, and 9 in forensic traces. These findings demonstrate that StealthCup elicits attacker behavior closely aligned with state-sponsored TTPs, while exposing blind spots across both open-source and commercial IDS. The resulting datasets and methodology provide a reproducible foundation for future stealth-focused IDS evaluation.

Related papers

• Who Pays the Price? Stakeholder-Centric Prompt Injection Benchmarking for Real-world Web Agents [93.19140872946842]
  Web agents driven by large language models (LLMs) are increasingly deployed in real-world environments.<n>This makes them vulnerable to prompt-injection attacks, in which seemingly benign content embeds adversarial instructions that manipulate agent behaviour.<n>We introduce textbfsysname, a benchmark to systematically categorize and attribute harm in real-world web agent systems.
  arXiv  Detail & Related papers  (2026-06-11T14:12:43Z)
• FlowGuard: Flow Matching for Identity-Independent Detection of Data-Free Model Stealing Attacks on Energy System Intrusion Detection Systems [0.0]
  FlowGuard is an identity-independent defence based on flow matching that classifies incoming queries as out-of-distribution (OOD)<n>This approach exploits the fact that queries generated synthetically for data-free model stealing attacks occupy a lower-dimensional manifold than real network traffic.<n>We evaluate our method against PRADA and FDINet using MAZE and DisGUIDE attacks in single-client and distributed (100-client Sybil) settings.
  arXiv  Detail & Related papers  (2026-06-02T10:18:45Z)
• UNAD+: An Explainable Hybrid Framework for Unknown Network Attack Detection [0.0]
  This paper presents an enhanced framework for unknown network attack detection derived from the previously proposed Unknown Network Attack Detector (UNAD)<n>UNAD+ combines a benign-only unsupervised ensemble with Weighted Majority Voting (WMV), a supervised refinement stage trained on pseudo-labelled detections, and a post hoc explainability layer.<n>The results show that UNAD+ improves on the original UNAD framework, achieving F1-scores above 98% across the benchmark datasets.
  arXiv  Detail & Related papers  (2026-05-21T15:31:10Z)
• Zero Day Attacks: Novel Behaviour or Novel Vulnerability? [0.0]
  Zero-day attacks pose severe cybersecurity risks due to their high success rates and stealth.<n>For an IDS to be effective it must be grounded in an understanding of how zero-day attacks manifest in real-world networks.<n>We propose a taxonomy of zero-day vulnerability types and analyze assumptions of ML-based intrusion detection approaches.
  arXiv  Detail & Related papers  (2026-05-04T20:24:30Z)
• StealthRL: Reinforcement Learning Paraphrase Attacks for Multi-Detector Evasion of AI-Text Detectors [0.0]
  StealthRL is a reinforcement learning framework for AI-text detectors.<n>We train a paraphrase policy against a multi-detector ensemble under realistic adversarial conditions.<n>StealthRL achieves near-zero detection, reduces mean AUROC from 0.74 to 0.27, and attains a 99.9% attack success rate.
  arXiv  Detail & Related papers  (2026-02-09T17:33:46Z)
• ReasAlign: Reasoning Enhanced Safety Alignment against Prompt Injection Attack [52.17935054046577]
  We present ReasAlign, a model-level solution to improve safety alignment against indirect prompt injection attacks.<n>ReasAlign incorporates structured reasoning steps to analyze user queries, detect conflicting instructions, and preserve the continuity of the user's intended tasks.
  arXiv  Detail & Related papers  (2026-01-15T08:23:38Z)
• A Hierarchical IDS for Zero-Day Attack Detection in Internet of Medical Things Networks [1.024113475677323]
  We propose a multi level IoMT IDS framework capable of detecting zero day attacks and distinguishing between known and unknown threats.<n>The first layer detects zero-day attacks with high accuracy without needing new datasets, ensuring strong applicability in IoMT environments.
  arXiv  Detail & Related papers  (2025-08-14T05:08:37Z)
• CANTXSec: A Deterministic Intrusion Detection and Prevention System for CAN Bus Monitoring ECU Activations [53.036288487863786]
  We propose CANTXSec, the first deterministic Intrusion Detection and Prevention system based on physical ECU activations.<n>It detects and prevents classical attacks in the CAN bus, while detecting advanced attacks that have been less investigated in the literature.<n>We prove the effectiveness of our solution on a physical testbed, where we achieve 100% detection accuracy in both classes of attacks while preventing 100% of FIAs.
  arXiv  Detail & Related papers  (2025-05-14T13:37:07Z)
• Evaluating the Robustness of Adversarial Defenses in Malware Detection Systems [2.209921757303168]
  We introduce a technique to convert continuous perturbations into binary feature spaces while preserving high attack success and low perturbation size.<n>Second, we present a novel adversarial method for binary domains, designed to achieve attack goals with minimal feature changes.<n> Experiments on the Malscan dataset show that sigma-binary outperforms existing attacks and exposes key vulnerabilities in state-of-the-art defenses.
  arXiv  Detail & Related papers  (2025-05-14T12:38:43Z)
• Adversarial Sample Generation for Anomaly Detection in Industrial Control Systems [2.6513941799808873]
  We generate adversarial samples using the Jacobian Saliency Map Attack (JSMA)<n>We validate the generalization and scalability of the adversarial samples to tackle a broad range of real attacks on Industrial Control Systems.<n>The model trained with adversarial samples detected attacks with 95% accuracy on real-world attack data not used during training.
  arXiv  Detail & Related papers  (2025-05-06T02:27:17Z)
• Defense against Joint Poison and Evasion Attacks: A Case Study of DERMS [2.632261166782093]
  We propose the first framework of IDS that is robust against joint poisoning and evasion attacks. We verify the robustness of our method on the IEEE-13 bus feeder model against a diverse set of poisoning and evasion attack scenarios.
  arXiv  Detail & Related papers  (2024-05-05T16:24:30Z)
• An Open Patch Generator based Fingerprint Presentation Attack Detection using Generative Adversarial Network [3.5558308387389626]
  Presentation Attack (PA) or spoofing is one of the threats caused by presenting a spoof of a genuine fingerprint to the sensor of Automatic Fingerprint Recognition Systems (AFRS) This paper proposes a CNN based technique that uses a Generative Adversarial Network (GAN) to augment the dataset with spoof samples generated from the proposed Open Patch Generator (OPG) An overall accuracy of 96.20%, 94.97%, and 92.90% has been achieved on the LivDet 2015, 2017, and 2019 databases, respectively under the LivDet protocol scenarios.
  arXiv  Detail & Related papers  (2023-06-06T10:52:06Z)
• DRSM: De-Randomized Smoothing on Malware Classifier Providing Certified Robustness [58.23214712926585]
  We develop a certified defense, DRSM (De-Randomized Smoothed MalConv), by redesigning the de-randomized smoothing technique for the domain of malware detection. Specifically, we propose a window ablation scheme to provably limit the impact of adversarial bytes while maximally preserving local structures of the executables. We are the first to offer certified robustness in the realm of static detection of malware executables.
  arXiv  Detail & Related papers  (2023-03-20T17:25:22Z)
• Adversarial Attacks and Defense for Non-Parametric Two-Sample Tests [73.32304304788838]
  This paper systematically uncovers the failure mode of non-parametric TSTs through adversarial attacks. To enable TST-agnostic attacks, we propose an ensemble attack framework that jointly minimizes the different types of test criteria. To robustify TSTs, we propose a max-min optimization that iteratively generates adversarial pairs to train the deep kernels.
  arXiv  Detail & Related papers  (2022-02-07T11:18:04Z)
• ADC: Adversarial attacks against object Detection that evade Context consistency checks [55.8459119462263]
  We show that even context consistency checks can be brittle to properly crafted adversarial examples. We propose an adaptive framework to generate examples that subvert such defenses. Our results suggest that how to robustly model context and check its consistency, is still an open problem.
  arXiv  Detail & Related papers  (2021-10-24T00:25:09Z)
• Evaluating the Robustness of Semantic Segmentation for Autonomous Driving against Real-World Adversarial Patch Attacks [62.87459235819762]
  In a real-world scenario like autonomous driving, more attention should be devoted to real-world adversarial examples (RWAEs) This paper presents an in-depth evaluation of the robustness of popular SS models by testing the effects of both digital and real-world adversarial patches.
  arXiv  Detail & Related papers  (2021-08-13T11:49:09Z)
• Policy Smoothing for Provably Robust Reinforcement Learning [109.90239627115336]
  We study the provable robustness of reinforcement learning against norm-bounded adversarial perturbations of the inputs. We generate certificates that guarantee that the total reward obtained by the smoothed policy will not fall below a certain threshold under a norm-bounded adversarial of perturbation the input.
  arXiv  Detail & Related papers  (2021-06-21T21:42:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.