CTI-HAL: A Human-Annotated Dataset for Cyber Threat Intelligence Analysis

• URL: http://arxiv.org/abs/2504.05866v1
• Date: Tue, 08 Apr 2025 09:47:15 GMT
• Title: CTI-HAL: A Human-Annotated Dataset for Cyber Threat Intelligence Analysis
• Authors: Sofia Della Penna, Roberto Natella, Vittorio Orbinato, Lorenzo Parracino, Luciano Pianese,
• Abstract summary: Cyber Threat Intelligence (CTI) sources are often unstructured and in natural language, making it difficult to automatically extract information.<n>Recent studies have explored the use of AI to perform automatic extraction from CTI data.<n>We introduce a novel dataset manually constructed from CTI reports and structured according to the MITRE ATT&CK framework.
• Score: 2.7862108332002546
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Organizations are increasingly targeted by Advanced Persistent Threats (APTs), which involve complex, multi-stage tactics and diverse techniques. Cyber Threat Intelligence (CTI) sources, such as incident reports and security blogs, provide valuable insights, but are often unstructured and in natural language, making it difficult to automatically extract information. Recent studies have explored the use of AI to perform automatic extraction from CTI data, leveraging existing CTI datasets for performance evaluation and fine-tuning. However, they present challenges and limitations that impact their effectiveness. To overcome these issues, we introduce a novel dataset manually constructed from CTI reports and structured according to the MITRE ATT&CK framework. To assess its quality, we conducted an inter-annotator agreement study using Krippendorff alpha, confirming its reliability. Furthermore, the dataset was used to evaluate a Large Language Model (LLM) in a real-world business context, showing promising generalizability.

Related papers

• AdvKT: An Adversarial Multi-Step Training Framework for Knowledge Tracing [64.79967583649407]
  Knowledge Tracing (KT) monitors students' knowledge states and simulates their responses to question sequences.<n>Existing KT models typically follow a single-step training paradigm, which leads to significant error accumulation.<n>We propose a novel Adversarial Multi-Step Training Framework for Knowledge Tracing (AdvKT) which focuses on the multi-step KT task.
  arXiv  Detail & Related papers  (2025-04-07T03:31:57Z)
• Towards a scalable AI-driven framework for data-independent Cyber Threat Intelligence Information Extraction [0.0]
  This paper introduces 0-CTI, a scalable AI-based framework designed for efficient CTI Information Extraction.<n>The proposed system processes complete text sequences of CTI reports to extract a cyber ontology of named entities and their relationships.<n>Our contribution is the development of 0-CTI, the first modular framework for CTI Information Extraction that supports both supervised and zero-shot learning.
  arXiv  Detail & Related papers  (2025-01-08T12:35:17Z)
• CTINEXUS: Leveraging Optimized LLM In-Context Learning for Constructing Cybersecurity Knowledge Graphs Under Data Scarcity [49.657358248788945]
  Textual descriptions in cyber threat intelligence (CTI) reports are rich sources of knowledge about cyber threats. Current CTI extraction methods lack flexibility and generalizability, often resulting in inaccurate and incomplete knowledge extraction. We propose CTINexus, a novel framework leveraging optimized in-context learning (ICL) of large language models.
  arXiv  Detail & Related papers  (2024-10-28T14:18:32Z)
• Actionable Cyber Threat Intelligence using Knowledge Graphs and Large Language Models [0.8192907805418583]
  Microsoft, Trend Micro, and CrowdStrike are using generative AI to facilitate CTI extraction. This paper addresses the challenge of automating the extraction of actionable CTI using advancements in Large Language Models (LLMs) and Knowledge Graphs (KGs) Our methodology evaluates techniques such as prompt engineering, the guidance framework, and fine-tuning to optimize information extraction and structuring. Experimental results demonstrate the effectiveness of our approach in extracting relevant information, with guidance and fine-tuning showing superior performance over prompt engineering.
  arXiv  Detail & Related papers  (2024-06-30T13:02:03Z)
• The Frontier of Data Erasure: Machine Unlearning for Large Language Models [56.26002631481726]
  Large Language Models (LLMs) are foundational to AI advancements. LLMs pose risks by potentially memorizing and disseminating sensitive, biased, or copyrighted information. Machine unlearning emerges as a cutting-edge solution to mitigate these concerns.
  arXiv  Detail & Related papers  (2024-03-23T09:26:15Z)
• Time for aCTIon: Automated Analysis of Cyber Threat Intelligence in the Wild [2.4669630540735215]
  Cyber Threat Intelligence (CTI) plays a crucial role in assessing risks and enhancing security for organizations. Existing tools for automated structured CTI extraction have performance limitations. We fill these gaps providing a new large open benchmark dataset and aCTIon, a structured CTI information extraction tool.
  arXiv  Detail & Related papers  (2023-07-14T13:43:16Z)
• ThreatKG: An AI-Powered System for Automated Open-Source Cyber Threat Intelligence Gathering and Management [65.0114141380651]
  ThreatKG is an automated system for OSCTI gathering and management. It efficiently collects a large number of OSCTI reports from multiple sources. It uses specialized AI-based techniques to extract high-quality knowledge about various threat entities.
  arXiv  Detail & Related papers  (2022-12-20T16:13:59Z)
• Automatic Mapping of Unstructured Cyber Threat Intelligence: An Experimental Study [1.1470070927586016]
  We present an experimental study on the automatic classification of unstructured Cyber Threat Intelligence (CTI) into attack techniques using machine learning (ML) We contribute with two new datasets for CTI analysis, and we evaluate several ML models, including both traditional and deep learning-based ones. We present several lessons learned about how ML can perform at this task, which classifiers perform best and under which conditions, which are the main causes of classification errors, and the challenges ahead for CTI analysis.
  arXiv  Detail & Related papers  (2022-08-25T15:01:42Z)
• Towards Automated Classification of Attackers' TTPs by combining NLP with ML Techniques [77.34726150561087]
  We evaluate and compare different Natural Language Processing (NLP) and machine learning techniques used for security information extraction in research. Based on our investigations we propose a data processing pipeline that automatically classifies unstructured text according to attackers' tactics and techniques.
  arXiv  Detail & Related papers  (2022-07-18T09:59:21Z)
• InfoBERT: Improving Robustness of Language Models from An Information Theoretic Perspective [84.78604733927887]
  Large-scale language models such as BERT have achieved state-of-the-art performance across a wide range of NLP tasks. Recent studies show that such BERT-based models are vulnerable facing the threats of textual adversarial attacks. We propose InfoBERT, a novel learning framework for robust fine-tuning of pre-trained language models.
  arXiv  Detail & Related papers  (2020-10-05T20:49:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.