SEvenLLM: Benchmarking, Eliciting, and Enhancing Abilities of Large Language Models in Cyber Threat Intelligence

• URL: http://arxiv.org/abs/2405.03446v2
• Date: Mon, 3 Jun 2024 04:04:52 GMT
• Title: SEvenLLM: Benchmarking, Eliciting, and Enhancing Abilities of Large Language Models in Cyber Threat Intelligence
• Authors: Hangyuan Ji, Jian Yang, Linzheng Chai, Chaoren Wei, Liqun Yang, Yunlong Duan, Yunli Wang, Tianzhen Sun, Hongcheng Guo, Tongliang Li, Changyu Ren, Zhoujun Li,
• Abstract summary: This paper introduces a framework to benchmark, elicit, and improve cybersecurity incident analysis and response abilities. We create a high-quality bilingual instruction corpus by crawling cybersecurity raw text from cybersecurity websites. The instruction dataset SEvenLLM-Instruct is used to train cybersecurity LLMs with the multi-task learning objective.
• Score: 27.550484938124193
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: To address the increasing complexity and frequency of cybersecurity incidents emphasized by the recent cybersecurity threat reports with over 10 billion instances, cyber threat intelligence (CTI) plays a critical role in the modern cybersecurity landscape by offering the insights required to understand and combat the constantly evolving nature of cyber threats. Inspired by the powerful capability of large language models (LLMs) in handling complex tasks, in this paper, we introduce a framework to benchmark, elicit, and improve cybersecurity incident analysis and response abilities in LLMs for Security Events (SEvenLLM). Specifically, we create a high-quality bilingual instruction corpus by crawling cybersecurity raw text from cybersecurity websites to overcome the lack of effective data for information extraction. Then, we design a pipeline to auto-select tasks from the tasks pool and convert the raw text into supervised corpora comprised of question and response. The instruction dataset SEvenLLM-Instruct is used to train cybersecurity LLMs with the multi-task learning objective (27 well-designed tasks) for augmenting the analysis of cybersecurity events. Extensive experiments in our curated benchmark (SEvenLLM-bench) demonstrate that SEvenLLM performs more sophisticated threat analysis and fortifies defenses against the evolving landscape of cyber threats.

Related papers

• CTINEXUS: Leveraging Optimized LLM In-Context Learning for Constructing Cybersecurity Knowledge Graphs Under Data Scarcity [49.657358248788945]
  Textual descriptions in cyber threat intelligence (CTI) reports are rich sources of knowledge about cyber threats. Current CTI extraction methods lack flexibility and generalizability, often resulting in inaccurate and incomplete knowledge extraction. We propose CTINexus, a novel framework leveraging optimized in-context learning (ICL) of large language models.
  arXiv  Detail & Related papers  (2024-10-28T14:18:32Z)
• A dataset for cyber threat intelligence modeling of connected autonomous vehicles [17.58243748365034]
  This paper reports the creation of a cyber threat intelligence corpus focusing on vehicle cybersecurity knowledge mining. The proposed dataset will serve as a valuable resource for evaluating the performance of existing algorithms and advancing research in cyber threat intelligence modeling within the automotive field.
  arXiv  Detail & Related papers  (2024-10-18T16:55:12Z)
• A Comprehensive Overview of Large Language Models (LLMs) for Cyber Defences: Opportunities and Directions [12.044950530380563]
  The recent progression of Large Language Models (LLMs) has witnessed great success in the fields of data-centric applications. We provide an overview for the recent activities of LLMs in cyber defence sections. Fundamental concepts of the progression of LLMs from Transformers, Pre-trained Transformers, and GPT is presented.
  arXiv  Detail & Related papers  (2024-05-23T12:19:07Z)
• Generative AI and Large Language Models for Cyber Security: All Insights You Need [0.06597195879147556]
  This paper provides a comprehensive review of the future of cybersecurity through Generative AI and Large Language Models (LLMs) We explore LLM applications across various domains, including hardware design security, intrusion detection, software engineering, design verification, cyber threat intelligence, malware detection, and phishing detection. We present an overview of LLM evolution and its current state, focusing on advancements in models such as GPT-4, GPT-3.5, Mixtral-8x7B, BERT, Falcon2, and LLaMA.
  arXiv  Detail & Related papers  (2024-05-21T13:02:27Z)
• Large Language Models for Cyber Security: A Systematic Literature Review [14.924782327303765]
  We conduct a comprehensive review of the literature on the application of Large Language Models in cybersecurity (LLM4Security) We observe that LLMs are being applied to a wide range of cybersecurity tasks, including vulnerability detection, malware analysis, network intrusion detection, and phishing detection. Third, we identify several promising techniques for adapting LLMs to specific cybersecurity domains, such as fine-tuning, transfer learning, and domain-specific pre-training.
  arXiv  Detail & Related papers  (2024-05-08T02:09:17Z)
• LLMs Killed the Script Kiddie: How Agents Supported by Large Language Models Change the Landscape of Network Threat Testing [4.899163798406851]
  We explore the potential of Large Language Models to reason about threats, generate information about tools, and automate cyber campaigns. We present prompt engineering approaches for a plan-act-report loop for one action of a threat campaign and and a prompt chaining design that directs the sequential decision process of a multi-action campaign.
  arXiv  Detail & Related papers  (2023-10-10T18:49:20Z)
• On the Security Risks of Knowledge Graph Reasoning [71.64027889145261]
  We systematize the security threats to KGR according to the adversary's objectives, knowledge, and attack vectors. We present ROAR, a new class of attacks that instantiate a variety of such threats. We explore potential countermeasures against ROAR, including filtering of potentially poisoning knowledge and training with adversarially augmented queries.
  arXiv  Detail & Related papers  (2023-05-03T18:47:42Z)
• Graph Mining for Cybersecurity: A Survey [61.505995908021525]
  The explosive growth of cyber attacks nowadays, such as malware, spam, and intrusions, caused severe consequences on society. Traditional Machine Learning (ML) based methods are extensively used in detecting cyber threats, but they hardly model the correlations between real-world cyber entities. With the proliferation of graph mining techniques, many researchers investigated these techniques for capturing correlations between cyber entities and achieving high performance.
  arXiv  Detail & Related papers  (2023-04-02T08:43:03Z)
• Towards Automated Classification of Attackers' TTPs by combining NLP with ML Techniques [77.34726150561087]
  We evaluate and compare different Natural Language Processing (NLP) and machine learning techniques used for security information extraction in research. Based on our investigations we propose a data processing pipeline that automatically classifies unstructured text according to attackers' tactics and techniques.
  arXiv  Detail & Related papers  (2022-07-18T09:59:21Z)
• A System for Efficiently Hunting for Cyber Threats in Computer Systems Using Threat Intelligence [78.23170229258162]
  We build ThreatRaptor, a system that facilitates cyber threat hunting in computer systems using OSCTI. ThreatRaptor provides (1) an unsupervised, light-weight, and accurate NLP pipeline that extracts structured threat behaviors from unstructured OSCTI text, (2) a concise and expressive domain-specific query language, TBQL, to hunt for malicious system activities, and (3) a query synthesis mechanism that automatically synthesizes a TBQL query from the extracted threat behaviors.
  arXiv  Detail & Related papers  (2021-01-17T19:44:09Z)
• Enabling Efficient Cyber Threat Hunting With Cyber Threat Intelligence [94.94833077653998]
  ThreatRaptor is a system that facilitates threat hunting in computer systems using open-source Cyber Threat Intelligence (OSCTI) It extracts structured threat behaviors from unstructured OSCTI text and uses a concise and expressive domain-specific query language, TBQL, to hunt for malicious system activities. Evaluations on a broad set of attack cases demonstrate the accuracy and efficiency of ThreatRaptor in practical threat hunting.
  arXiv  Detail & Related papers  (2020-10-26T14:54:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.