A Structured Cyber Threat Intelligence Dataset Using STIX 2.1 Entities and MITRE ATT&CK Mappings

• URL: http://arxiv.org/abs/2607.23312v1
• Date: Sat, 25 Jul 2026 17:59:35 GMT
• Title: A Structured Cyber Threat Intelligence Dataset Using STIX 2.1 Entities and MITRE ATT&CK Mappings
• Abstract summary: This study presents a manually constructed dataset of 150 English-language CTI reports.<n>Report-level datasets that preserve complex entity relationships and normalized adversarial behaviors remain limited.<n>The dataset provides a benchmark for CTI information extraction, knowledge-graph construction, incident analysis, and threat attribution.
• Score: 0.20415910628419062
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Cyber threat intelligence (CTI) reports are typically written in unstructured formats, which complicates the extraction and analysis of important entities and adversarial behaviors. Although existing CTI research provides extraction tools, knowledge-graph frameworks, and MITRE ATT&CK mapped datasets, curated report-level datasets that preserve complex entity relationships and normalized adversarial behaviors remain limited. To address this limitation, this study presents a manually constructed dataset of 150 English-language CTI reports, each represented as STIX 2.1 based graphs, which includes 4,777 STIX entities, 5,817 STIX relationships in total, and 1,273 STIX attack-pattern entities (adversarial behaviors) mapped to 269 unique MITRE ATT&CK Enterprise techniques and sub-techniques. Twenty five randomly sampled reports were independently assessed by two cybersecurity researchers, which shows substantial inter-rater agreement. Disagreements were subsequently adjudicated to establish a gold-standard reference dataset. Four locally deployed open-source LLMs were evaluated as automated judges against this adjudicated reference sample. Qwen3.6:27B achieved the strongest overall performance, with a maximum kappa score of 0.803, micro-F1 scores exceeding 92%, and false-positive rates below 5%. The dataset provides a benchmark for CTI information extraction, knowledge-graph construction, incident analysis, and threat attribution. The findings further indicate that locally deployed LLMs can support human reviewers in identifying annotation inconsistencies, but expert validation remains essential.

Related papers

• Evaluating Open-Source LLMs for Multi-Label ATT&CK Technique Classification on CTI Reports [3.7098231493739764]
  Classifying Cyber Threat Intelligence (CTI) using Adversarial Tactics, Techniques, and Common Knowledge (ATT&CK) is essential for proactive defense, but historically required extensive human effort.<n>Pre-Large Language Model (LLM) automation sped up this process, but could not resolve the complex language and multi-step attack patterns found in unstructured CTI reports.<n>LLMs addressed previous limitations by using contextual reasoning to understand unstructured text.<n>Current evaluations rely on simplified, single-technique sentences that ignore the complexity of real-world CTI reports, which often leads to inflated performance results.
  arXiv  Detail & Related papers  (2026-06-16T17:04:15Z)
• Context-aware Entity-Relation Extraction for Threat Intelligence Knowledge Graphs [0.7591490481106252]
  Cybersecurity Knowledge Graphs (CKGs) unify diverse Cyber Threat Intelligence (CTI) sources into structured, queryable formats.<n>Current pipeline-based approaches often suffer from error propagation, reducing extraction accuracy and limiting generalizability.<n>This paper introduces the Context-aware Threat Intelligence Knowledge Graph (CTiKG) framework, a pipeline architecture designed to accurately extract and classify threat entities.
  arXiv  Detail & Related papers  (2026-05-15T12:42:47Z)
• DataClaw: A Process-Oriented Agent Benchmark for Exploratory Real-World Data Analysis [76.98578575566184]
  DataClaw is a process-oriented benchmark for exploratory real-world data analysis.<n>It contains approximately 2.06 million real-world records across enterprise, industry and policy domains.<n>DataClaw measures how far an agent progresses and where its reasoning breaks down.
  arXiv  Detail & Related papers  (2026-05-04T11:57:09Z)
• Encyclo-K: Evaluating LLMs with Dynamically Composed Knowledge Statements [78.87065404966002]
  Existing benchmarks predominantly curate questions at the question level.<n>We propose Encyclo-K, a statement-based benchmark that rethinks benchmark construction from the ground up.
  arXiv  Detail & Related papers  (2025-12-31T13:55:54Z)
• AutoMalDesc: Large-Scale Script Analysis for Cyber Threat Research [81.04845910798387]
  Generating natural language explanations for threat detections remains an open problem in cybersecurity research.<n>We present AutoMalDesc, an automated static analysis summarization framework that operates independently at scale.<n>We publish our complete dataset of more than 100K script samples, including annotated seed (0.9K) datasets, along with our methodology and evaluation framework.
  arXiv  Detail & Related papers  (2025-11-17T13:05:25Z)
• From Text to Actionable Intelligence: Automating STIX Entity and Relationship Extraction [8.623367082899946]
  AZERG is a tool designed to assist security analysts in automatically generating structured STIX representations.<n>We apply task-specific fine-tuning to accurately extract relevant entities and infer their relationships in accordance with the STIX specification.<n>Our models achieved F1-scores of 84.43% for T1, 88.49% for T2, 95.47% for T3, and 84.60% for T4 in real-world scenarios.
  arXiv  Detail & Related papers  (2025-07-22T13:27:09Z)
• FORGE: An LLM-driven Framework for Large-Scale Smart Contract Vulnerability Dataset Construction [34.20628333535654]
  FORGE is the first automated approach for constructing smart contract vulnerability datasets.<n>We generate a dataset comprising 81,390 solidity files and 27,497 vulnerability findings across 296 CWE categories.<n>Results reveal the significant limitations in current detection capabilities.
  arXiv  Detail & Related papers  (2025-06-23T16:03:16Z)
• CTI-HAL: A Human-Annotated Dataset for Cyber Threat Intelligence Analysis [2.7862108332002546]
  Cyber Threat Intelligence (CTI) sources are often unstructured and in natural language, making it difficult to automatically extract information.<n>Recent studies have explored the use of AI to perform automatic extraction from CTI data.<n>We introduce a novel dataset manually constructed from CTI reports and structured according to the MITRE ATT&CK framework.
  arXiv  Detail & Related papers  (2025-04-08T09:47:15Z)
• SciER: An Entity and Relation Extraction Dataset for Datasets, Methods, and Tasks in Scientific Documents [49.54155332262579]
  We release a new entity and relation extraction dataset for entities related to datasets, methods, and tasks in scientific articles. Our dataset contains 106 manually annotated full-text scientific publications with over 24k entities and 12k relations.
  arXiv  Detail & Related papers  (2024-10-28T15:56:49Z)
• DiscoveryBench: Towards Data-Driven Discovery with Large Language Models [50.36636396660163]
  We present DiscoveryBench, the first comprehensive benchmark that formalizes the multi-step process of data-driven discovery. Our benchmark contains 264 tasks collected across 6 diverse domains, such as sociology and engineering. Our benchmark, thus, illustrates the challenges in autonomous data-driven discovery and serves as a valuable resource for the community to make progress.
  arXiv  Detail & Related papers  (2024-07-01T18:58:22Z)
• Exploiting Contextual Target Attributes for Target Sentiment Classification [53.30511968323911]
  Existing PTLM-based models for TSC can be categorized into two groups: 1) fine-tuning-based models that adopt PTLM as the context encoder; 2) prompting-based models that transfer the classification task to the text/word generation task. We present a new perspective of leveraging PTLM for TSC: simultaneously leveraging the merits of both language modeling and explicit target-context interactions via contextual target attributes.
  arXiv  Detail & Related papers  (2023-12-21T11:45:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.