GenTI: Benchmarking LLMs for Autonomous IDPS Rule Generation for Unseen Attacks

• URL: http://arxiv.org/abs/2606.05844v1
• Date: Thu, 04 Jun 2026 08:19:52 GMT
• Title: GenTI: Benchmarking LLMs for Autonomous IDPS Rule Generation for Unseen Attacks
• Abstract summary: Generative Thread Intelligence (GenTI) is a benchmark for automatic generation of IDPS rules targeting unseen attacks.<n>GTI aggregates over 150k detection and prevention rules from Snort, Suricata, Emerging Threats, as well as 50k YARA, each annotated with protocol behavior, payload signatures, contextual relationships, mappings to Cyber Threat Intelligence (CTI)<n>Our GenTI instantiation achieves a composite rule-quality score of 89.4%, with 94.8% CTI coverage, improving unseen attacks detection from 45% to 87.4% and reducing the false-positive rate from 8.5%
• Score: 1.7816843507516946
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Rule-based Intrusion Detection and Prevention Systems (IDPS) offer precise attack detection as well as mitigation, however their manually crafted, signature-driven rules limit adaptability to emerging and zero-day threats. Additionally, existing public datasets (e.g., CICIDS2017, UNSW-NB15) focus on traffic classification and provide little structured information to support automatic rule synthesis or prevention logic. To address this gap, we propose Generative Thread Intelligence (GenTI) \footnote{GenTI refers to the proposed framework, and GTI refers to the dataset.} an LLM-driven benchmark for automatic generation of IDPS rules targeting unseen attacks. The dataset (GTI) aggregates over 150k detection and prevention rules from Snort, Suricata, Emerging Threats, as well as 50k YARA, each annotated with protocol behavior, payload signatures, contextual relationships, mappings to Cyber Threat Intelligence (CTI), along with actionable response types (alert, drop, reject). Moreover, on top of this corpus we design an LLM-based pipeline that transforms analyst prompts and representative payloads into deployable rules via structured prompt engineering, Chain-of-Thought (CoT) reasoning, as well as a Chain-of-Verification (CoVe) loop for syntactic, semantic, and security validation. The generated rules are executed in real time on (Snort/Suricata) and evaluated by syntax accuracy, semantic similarity, CTI coverage, security effectiveness as well as unseen attacks detection. Furthermore, our GenTI instantiation achieves a composite rule-quality score of 89.4\%, with 94.8\% CTI coverage, improving unseen attacks detection from 45\% to 87.4\% and reducing the false-positive rate from 8.5\% to 2.3\%. Overall, GenTI establishes the first large-scale benchmark that tightly couples rule-level CTI with LLM-based automation, enabling adaptive, self-evolving IDPS.

Related papers

• An Automated Framework for Extracting Reachable Attack Chains from Cyber Threat Intelligence Reports [8.445810236675674]
  Cyber Threat Intelligence (CTI) reports richly describe real-world attack processes, but their unstructured narratives cannot be directly used for automated attack-path reasoning.<n>This paper proposes an automated framework that extracts reachable attack chains by modeling each attack step as an attack unit of preconditions, an attack behavior, and postconditions.
  arXiv  Detail & Related papers  (2026-07-22T04:16:35Z)
• From Attack Simulation to SIEM Rule: Deterministic Detection-as-Code Synthesis with Probe-Level Traceability [51.56484100374058]
  Security teams simulate attacks against their own systems to check whether their monitoring would catch a real intruder.<n>A human bridges that gap by hand, reading each finding and writing the corresponding Sigma rule.<n>We show this translation can be partially automated when probes are drawn from a locked corpus.
  arXiv  Detail & Related papers  (2026-06-03T14:26:25Z)
• SCI-Defense: Defending Manipulation Attacks from Generative Engine Optimization [30.491360390710422]
  We propose SCI-Defense, a three-component defense framework combining Perplexity detection, Semantic Integrity Scoring, and Inter-Candidate Detection.<n>On 600 MS MARCO web passages, String attacks are blocked with perfect recall while Review attacks yield near-zero recall, as web passages lack the persuasion-oriented signals that SIS targets in product descriptions.
  arXiv  Detail & Related papers  (2026-05-21T03:28:06Z)
• TwinGate: Stateful Defense against Decompositional Jailbreaks in Untraceable Traffic via Asymmetric Contrastive Learning [60.68349524623048]
  Decompositional jailbreaks pose a critical threat to large language models.<n>We present TwinGate, a stateful dual-encoder defense framework.<n>We construct a comprehensive dataset of over 3.62 million instructions spanning 8,600 distinct malicious intents.
  arXiv  Detail & Related papers  (2026-04-30T13:44:01Z)
• T-MAP: Red-Teaming LLM Agents with Trajectory-aware Evolutionary Search [49.99459363244884]
  We propose a trajectory-aware evolutionary search method, T-MAP, which leverages execution trajectories to guide the discovery of adversarial prompts.<n>Our approach enables the automatic generation of attacks that not only bypass safety guardrails but also reliably realize harmful objectives through actual tool interactions.
  arXiv  Detail & Related papers  (2026-03-21T12:33:34Z)
• Autonomous Chain-of-Thought Distillation for Graph-Based Fraud Detection [73.9189065770752]
  Graph-based fraud detection on text-attributed graphs (TAGs) requires jointly modeling rich textual semantics and relational dependencies.<n>We propose FraudCoT, a unified framework that advances TAG-based fraud detection through autonomous, graph-aware chain-of-thought (CoT) reasoning and scalable LLM-GNN co-training.
  arXiv  Detail & Related papers  (2026-01-30T13:12:12Z)
• The Trojan Knowledge: Bypassing Commercial LLM Guardrails via Harmless Prompt Weaving and Adaptive Tree Search [58.8834056209347]
  Large language models (LLMs) remain vulnerable to jailbreak attacks that bypass safety guardrails to elicit harmful outputs.<n>We introduce the Correlated Knowledge Attack Agent (CKA-Agent), a dynamic framework that reframes jailbreaking as an adaptive, tree-structured exploration of the target model's knowledge base.
  arXiv  Detail & Related papers  (2025-12-01T07:05:23Z)
• GRIDAI: Generating and Repairing Intrusion Detection Rules via Collaboration among Multiple LLM-based Agents [16.185318922819466]
  GRIDAI is a novel end-to-end framework for the automated Generation and Repair of Intrusion Detection rules.<n> GRIDAI identifies the relationships between new attack samples and existing rules, efficiently generates and repairs rules to handle new attacks and variants.
  arXiv  Detail & Related papers  (2025-10-15T08:05:38Z)
• FALCON: Autonomous Cyber Threat Intelligence Mining with LLMs for IDS Rule Generation [4.432089452255636]
  Signature-based Intrusion Detection Systems (IDS) detect malicious activities by matching network or host activity against predefined rules.<n>Recent advancements in agentic systems powered by Large Language Models (LLMs) offer the potential for autonomous IDS rule generation with internal evaluation.<n>We introduce FALCON, an autonomous agentic framework that generates deployable IDS rules from CTI data in real-time and evaluates them using built-in multi-phased validators.
  arXiv  Detail & Related papers  (2025-08-26T05:08:53Z)
• CyberRAG: An Agentic RAG cyber attack classification and reporting tool [0.3914676152740142]
  CyberRAG is a modular agent-based RAG framework that delivers real-time classification, explanation, and structured reporting for cyber-attacks.<n>Unlike traditional RAG, CyberRAG adopts an agentic design that enables dynamic control flow and adaptive reasoning.
  arXiv  Detail & Related papers  (2025-07-03T08:32:19Z)
• AegisLLM: Scaling Agentic Systems for Self-Reflective Defense in LLM Security [74.22452069013289]
  AegisLLM is a cooperative multi-agent defense against adversarial attacks and information leakage.<n>We show that scaling agentic reasoning system at test-time substantially enhances robustness without compromising model utility.<n> Comprehensive evaluations across key threat scenarios, including unlearning and jailbreaking, demonstrate the effectiveness of AegisLLM.
  arXiv  Detail & Related papers  (2025-04-29T17:36:05Z)
• Towards Copyright Protection for Knowledge Bases of Retrieval-augmented Language Models via Reasoning [58.57194301645823]
  Large language models (LLMs) are increasingly integrated into real-world personalized applications.<n>The valuable and often proprietary nature of the knowledge bases used in RAG introduces the risk of unauthorized usage by adversaries.<n>Existing methods that can be generalized as watermarking techniques to protect these knowledge bases typically involve poisoning or backdoor attacks.<n>We propose name for harmless' copyright protection of knowledge bases.
  arXiv  Detail & Related papers  (2025-02-10T09:15:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.