From Text to Actionable Intelligence: Automating STIX Entity and Relationship Extraction

• URL: http://arxiv.org/abs/2507.16576v1
• Date: Tue, 22 Jul 2025 13:27:09 GMT
• Title: From Text to Actionable Intelligence: Automating STIX Entity and Relationship Extraction
• Abstract summary: AZERG is a tool designed to assist security analysts in automatically generating structured STIX representations.<n>We apply task-specific fine-tuning to accurately extract relevant entities and infer their relationships in accordance with the STIX specification.<n>Our models achieved F1-scores of 84.43% for T1, 88.49% for T2, 95.47% for T3, and 84.60% for T4 in real-world scenarios.
• Score: 8.623367082899946
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Sharing methods of attack and their effectiveness is a cornerstone of building robust defensive systems. Threat analysis reports, produced by various individuals and organizations, play a critical role in supporting security operations and combating emerging threats. To enhance the timeliness and automation of threat intelligence sharing, several standards have been established, with the Structured Threat Information Expression (STIX) framework emerging as one of the most widely adopted. However, generating STIX-compatible data from unstructured security text remains a largely manual, expert-driven process. To address this challenge, we introduce AZERG, a tool designed to assist security analysts in automatically generating structured STIX representations. To achieve this, we adapt general-purpose large language models for the specific task of extracting STIX-formatted threat data. To manage the complexity, the task is divided into four subtasks: entity detection (T1), entity type identification (T2), related pair detection (T3), and relationship type identification (T4). We apply task-specific fine-tuning to accurately extract relevant entities and infer their relationships in accordance with the STIX specification. To address the lack of training data, we compiled a comprehensive dataset with 4,011 entities and 2,075 relationships extracted from 141 full threat analysis reports, all annotated in alignment with the STIX standard. Our models achieved F1-scores of 84.43% for T1, 88.49% for T2, 95.47% for T3, and 84.60% for T4 in real-world scenarios. We validated their performance against a range of open- and closed-parameter models, as well as state-of-the-art methods, demonstrating improvements of 2-25% across tasks.

Related papers

• SkillsMetric: Mapping the Detection Boundary of Static Analysis for Malicious Agent Skills [12.451393173022344]
  We present textscSkillsMetric, a five-stage static analysis framework that scores skill packages along pattern density, statistical anomaly, dataflow taint, import anomaly, and capability mismatch dimensions.<n>We construct an adversarial evaluation dataset of 2266 skills spanning 16attack types across code-level, system-level, and semantic-level threats, and evaluate on the full SkillMD-138K corpus.<n>Our framework achieves an AUC of 0.93 and 5-fold cross-validated F1 of 73.4%$pm$0.5%, with strong detection of data ex
  arXiv  Detail & Related papers  (2026-08-09T04:19:10Z)
• A Structured Cyber Threat Intelligence Dataset Using STIX 2.1 Entities and MITRE ATT&CK Mappings [0.20415910628419062]
  This study presents a manually constructed dataset of 150 English-language CTI reports.<n>Report-level datasets that preserve complex entity relationships and normalized adversarial behaviors remain limited.<n>The dataset provides a benchmark for CTI information extraction, knowledge-graph construction, incident analysis, and threat attribution.
  arXiv  Detail & Related papers  (2026-07-25T17:59:35Z)
• Evaluating Open-Weight LLMs for Generating Structured Threat Information for Autonomous Vehicle Vulnerabilities [0.30915521808748864]
  We evaluate open-weight Large Language Models (LLMs) for generating Structured Threat Information Expression (STIX)<n>We construct a dataset called CAV-STIX that maps vulnerability descriptions to STIX domain objects (SDO), STIX objects (SRO), Common Weaknession (CWE), and MITRE ATT&CK techniques mappings.
  arXiv  Detail & Related papers  (2026-07-17T17:55:19Z)
• Context-aware Entity-Relation Extraction for Threat Intelligence Knowledge Graphs [0.7591490481106252]
  Cybersecurity Knowledge Graphs (CKGs) unify diverse Cyber Threat Intelligence (CTI) sources into structured, queryable formats.<n>Current pipeline-based approaches often suffer from error propagation, reducing extraction accuracy and limiting generalizability.<n>This paper introduces the Context-aware Threat Intelligence Knowledge Graph (CTiKG) framework, a pipeline architecture designed to accurately extract and classify threat entities.
  arXiv  Detail & Related papers  (2026-05-15T12:42:47Z)
• TIJERE: A Novel Threat Intelligence Joint Extraction Model Based on Analyst Expert Knowledge [0.7591490481106252]
  TIJERE is an innovative joint entity and relation extraction framework.<n>This paper also introduces DNRTI-JE, the first publicly available jointly labeled dataset for cybersecurity entity and relation extraction.<n> TIJERE achieves state-of-the-art performance, with F1-scores exceeding 0.93 for NER and 0.98 for RE, outperforming existing methods.
  arXiv  Detail & Related papers  (2026-05-03T20:10:02Z)
• From Insight to Action: A Novel Framework for Interpretability-Guided Data Selection in Large Language Models [73.72877445629383]
  Interpretability-Guided Data Selection (IGDS) is a framework that first identifies causal task features through frequency recall and interventional filtering.<n>We validate IGDS on mathematical reasoning, summarization, and translation tasks within Gemma-2, LLaMA-3.1, and Qwen3 models.
  arXiv  Detail & Related papers  (2026-04-28T03:16:24Z)
• Temporal Attack Pattern Detection in Multi-Agent AI Workflows: An Open Framework for Training Trace-Based Security Models [0.0]
  We present an openly documented methodology for fine-tuning language models to detect temporal attack patterns in multi-agent AI.<n>We curate a dataset of 80,851 examples from 18 public cybersecurity sources and 35,026 synthetic OpenTelemetry traces.<n>Our custom benchmark accuracy improves from 42.86% to 74.29%, a statistically significant 31.4-point gain.
  arXiv  Detail & Related papers  (2025-12-29T09:41:22Z)
• Text Adversarial Attacks with Dynamic Outputs [63.25160580569707]
  Text adversarial attack methods are typically designed for static scenarios with fixed numbers of output labels and a predefined label space.<n>We introduce the Textual Dynamic Outputs Attack (TDOA) method, which employs a clustering-based surrogate model training approach.<n>With a single query per text, TDOA achieves a maximum attack success rate of 50.81%.<n>We extend TDOA to generative settings, surpassing prior results by up to 0.64 RDBLEU and 0.62 RDchrF.
  arXiv  Detail & Related papers  (2025-09-26T14:21:46Z)
• OmniEAR: Benchmarking Agent Reasoning in Embodied Tasks [52.87238755666243]
  We present OmniEAR, a framework for evaluating how language models reason about physical interactions, tool usage, and multi-agent coordination in embodied tasks.<n>We model continuous physical properties and complex spatial relationships across 1,500 scenarios spanning household and industrial domains.<n>Our systematic evaluation reveals severe performance degradation when models must reason from constraints.
  arXiv  Detail & Related papers  (2025-08-07T17:54:15Z)
• FORGE: An LLM-driven Framework for Large-Scale Smart Contract Vulnerability Dataset Construction [34.20628333535654]
  FORGE is the first automated approach for constructing smart contract vulnerability datasets.<n>We generate a dataset comprising 81,390 solidity files and 27,497 vulnerability findings across 296 CWE categories.<n>Results reveal the significant limitations in current detection capabilities.
  arXiv  Detail & Related papers  (2025-06-23T16:03:16Z)
• OVERT: A Benchmark for Over-Refusal Evaluation on Text-to-Image Models [73.6716695218951]
  Over-refusal is a phenomenon known as $textitover-refusal$ that reduces the practical utility of T2I models.<n>We present OVERT ($textbfOVE$r-$textbfR$efusal evaluation on $textbfT$ext-to-image models), the first large-scale benchmark for assessing over-refusal behaviors.
  arXiv  Detail & Related papers  (2025-05-27T15:42:46Z)
• Towards Effective Identification of Attack Techniques in Cyber Threat Intelligence Reports using Large Language Models [5.304267859042463]
  This work evaluates the performance of Cyber Threat Intelligence (CTI) extraction methods in identifying attack techniques from threat reports available on the web.<n>We analyse four configurations utilising state-of-the-art tools, including the Threat Report ATT&CK Mapper (TRAM) and open-source Large Language Models (LLMs) such as Llama2.<n>Our findings reveal significant challenges, including class imbalance, overfitting, and domain-specific complexity, which impede accurate technique extraction.
  arXiv  Detail & Related papers  (2025-05-06T03:43:12Z)
• CTI-HAL: A Human-Annotated Dataset for Cyber Threat Intelligence Analysis [2.7862108332002546]
  Cyber Threat Intelligence (CTI) sources are often unstructured and in natural language, making it difficult to automatically extract information.<n>Recent studies have explored the use of AI to perform automatic extraction from CTI data.<n>We introduce a novel dataset manually constructed from CTI reports and structured according to the MITRE ATT&CK framework.
  arXiv  Detail & Related papers  (2025-04-08T09:47:15Z)
• OmniParser V2: Structured-Points-of-Thought for Unified Visual Text Parsing and Its Generality to Multimodal Large Language Models [58.45517851437422]
  Visually-situated text parsing (VsTP) has recently seen notable advancements, driven by the growing demand for automated document understanding.<n>Existing solutions often rely on task-specific architectures and objectives for individual tasks.<n>In this paper, we introduce Omni V2, a universal model that unifies VsTP typical tasks, including text spotting, key information extraction, table recognition, and layout analysis.
  arXiv  Detail & Related papers  (2025-02-22T09:32:01Z)
• CTINexus: Automatic Cyber Threat Intelligence Knowledge Graph Construction Using Large Language Models [49.657358248788945]
  Textual descriptions in cyber threat intelligence (CTI) reports are rich sources of knowledge about cyber threats.<n>Current CTI knowledge extraction methods lack flexibility and generalizability.<n>We propose CTINexus, a novel framework for data-efficient CTI knowledge extraction and high-quality cybersecurity knowledge graph (CSKG) construction.
  arXiv  Detail & Related papers  (2024-10-28T14:18:32Z)
• TTPXHunter: Actionable Threat Intelligence Extraction as TTPs from Finished Cyber Threat Reports [3.2183320563774833]
  Knowing the modus operandi of adversaries aids organizations in employing efficient defensive strategies and sharing intelligence in the community. A translation tool is needed to interpret the modus operandi explained in the sentences of the threat report and translate it into a structured format. This research introduces a methodology named TTPXHunter for the automated extraction of threat intelligence in terms of Tactics, Techniques, and Procedures (TTPs) from finished cyber threat reports.
  arXiv  Detail & Related papers  (2024-03-05T19:04:09Z)
• TAT-LLM: A Specialized Language Model for Discrete Reasoning over Tabular and Textual Data [73.29220562541204]
  We consider harnessing the amazing power of language models (LLMs) to solve our task. We develop a TAT-LLM language model by fine-tuning LLaMA 2 with the training data generated automatically from existing expert-annotated datasets.
  arXiv  Detail & Related papers  (2024-01-24T04:28:50Z)
• Exploiting Contextual Target Attributes for Target Sentiment Classification [53.30511968323911]
  Existing PTLM-based models for TSC can be categorized into two groups: 1) fine-tuning-based models that adopt PTLM as the context encoder; 2) prompting-based models that transfer the classification task to the text/word generation task. We present a new perspective of leveraging PTLM for TSC: simultaneously leveraging the merits of both language modeling and explicit target-context interactions via contextual target attributes.
  arXiv  Detail & Related papers  (2023-12-21T11:45:28Z)
• AGIR: Automating Cyber Threat Intelligence Reporting with Natural Language Generation [15.43868945929965]
  We introduce AGIR (Automatic Generation of Intelligence Reports), a transformative tool for CTI reporting. AGIR's primary objective is to empower security analysts by automating the labor-intensive task of generating comprehensive intelligence reports. We evaluate AGIR's report generation capabilities both quantitatively and qualitatively.
  arXiv  Detail & Related papers  (2023-10-04T08:25:37Z)
• Alert-ME: An Explainability-Driven Defense Against Adversarial Examples in Transformer-Based Text Classification [9.818997495801705]
  This paper presents a unified framework called Explainability-driven Detection, Identification, and Transformation (EDIT) to strengthen inference-time defenses.<n>EDIT integrates explainability tools, including attention maps and integrated gradients, with frequency-based features to automatically detect and identify adversarial perturbations.<n>The framework provides robust, interpretable, and efficient protection against both standard, zero-day, and adaptive adversarial threats in text classification models.
  arXiv  Detail & Related papers  (2023-07-03T03:17:20Z)
• Automated Few-shot Classification with Instruction-Finetuned Language Models [76.69064714392165]
  We show that AuT-Few outperforms state-of-the-art few-shot learning methods. We also show that AuT-Few is the best ranking method across datasets on the RAFT few-shot benchmark.
  arXiv  Detail & Related papers  (2023-05-21T21:50:27Z)
• Text Classification via Large Language Models [63.1874290788797]
  We introduce Clue And Reasoning Prompting (CARP) to address complex linguistic phenomena involved in text classification. Remarkably, CARP yields new SOTA performances on 4 out of 5 widely-used text-classification benchmarks. More importantly, we find that CARP delivers impressive abilities on low-resource and domain-adaptation setups.
  arXiv  Detail & Related papers  (2023-05-15T06:24:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.