Toward Cybersecurity-Expert Small Language Models

• URL: http://arxiv.org/abs/2510.14113v1
• Date: Wed, 15 Oct 2025 21:34:58 GMT
• Title: Toward Cybersecurity-Expert Small Language Models
• Abstract summary: We present CyberPal 2.0, a family of cybersecurity-expert small language models (SLMs) ranging from 4B-20B parameters.<n>We generate an enriched chain-of-thought cybersecurity instruction dataset built with our data enrichment and formatting pipeline, SecKnowledge 2.0.<n>Across diverse cybersecurity benchmarks, CyberPal 2.0 consistently outperforms its baselines and surpasses various open and closed-source frontier models.
• Score: 3.155658695525581
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models (LLMs) are transforming everyday applications, yet deployment in cybersecurity lags due to a lack of high-quality, domain-specific models and training datasets. To address this gap, we present CyberPal 2.0, a family of cybersecurity-expert small language models (SLMs) ranging from 4B-20B parameters. To train CyberPal 2.0, we generate an enriched chain-of-thought cybersecurity instruction dataset built with our data enrichment and formatting pipeline, SecKnowledge 2.0, which integrates expert-in-the-loop steering of reasoning formats alongside LLM-driven multi-step grounding, yielding higher-fidelity, task-grounded reasoning traces for security tasks. Across diverse cybersecurity benchmarks, CyberPal 2.0 consistently outperforms its baselines and matches or surpasses various open and closed-source frontier models, while remaining a fraction of their size. On core cyber threat intelligence knowledge tasks, our models outperform almost all tested frontier models, ranking second only to Sec-Gemini v1. On core threat-investigation tasks, such as correlating vulnerabilities and bug tickets with weaknesses, our best 20B-parameter model outperforms GPT-4o, o1, o3-mini, and Sec-Gemini v1, ranking first, while our smallest 4B-parameter model ranks second.

Related papers

• CyberFactory: Scaling Cyber Security Capabilities with Instances from the Wild [49.44069582581181]
  CyberFactory transforms public vulnerability artifacts, including CVEs from the wild, into executable and verifiable task instances.<n>It uses a reusable vulnerability-analysis skill to guide the teacher through source inspection, problem solving with domain prior, and evidence-based validation.<n>On CyberGym, modelname reaches 52.4% Pass@1 under a one-hour budget, improving over its Qwen3.5 base model by +22.8 points.
  arXiv  Detail & Related papers  (2026-08-24T12:30:10Z)
• Embodied-R1.5: Evolving Physical Intelligence via Embodied Foundation Models [63.96252564303143]
  Embodied-R1.5 is a unified Embodied Foundation Model (EFM) that integrates comprehensive embodied reasoning capabilities.<n>We build a large-scale data system of over 15B tokens, and design a multi-task balanced RL recipe.<n>With only 8B parameters, Embodied-R1.5 SOTA on 16 out of 24 embodied VLM benchmarks, surpassing leading models like Gemini-Robotics-ER-1.5 and GPT-5.4.
  arXiv  Detail & Related papers  (2026-06-09T18:07:50Z)
• Are Frontier LLMs Ready for Cybersecurity? Evidence for Vertical Foundation Models from Dual-Mode Vulnerability Benchmarks [0.3303672705634661]
  We evaluate whether frontier LLMs are ready for cybersecurity through a dual-mode benchmark.<n>We test six frontier models (GPT-5.4, Codex5.3, Claude Opus4.6, Sonnet4.6, Gemini3.1Pro and Gemini3Flash) and two domain-specialized models across four testing paradigms.
  arXiv  Detail & Related papers  (2026-05-22T05:24:43Z)
• Operationalising Cyber Risk Management Using AI: Connecting Cyber Incidents to MITRE ATT&CK Techniques, Security Controls, and Metrics [0.0]
  This research presents a novel framework that leverages Natural Language Processing to address these challenges.<n>We introduce the Cyber Catalog, a knowledge base that integrates CIS Critical Security Controls, MITRE ATT&CK techniques, and SMART metrics.<n>To operationalise the framework, we fine-tuned all-mpnet-base-v2, a highly regarded sentence-transformers model used to convert text into numerical vectors.
  arXiv  Detail & Related papers  (2026-03-12T21:12:53Z)
• MagicAgent: Towards Generalized Agent Planning [73.21129030631421]
  We present textbfMagicAgent, a series of foundation models specifically designed for generalized agent planning.<n>We introduce a lightweight and scalable synthetic data framework that generates high-quality trajectories across diverse planning tasks.<n>We show that MagicAgent-32B and MagicAgent-30B-A3B achieve superior performance across diverse open-source benchmarks.
  arXiv  Detail & Related papers  (2026-02-22T01:39:16Z)
• AgentCPM-Explore: Realizing Long-Horizon Deep Exploration for Edge-Scale Agents [75.67445299298949]
  AgentCPM-Explore is a compact 4B agent model with high knowledge density and strong exploration capability.<n>We introduce a holistic training framework featuring parameter-space model fusion, reward signal denoising, and contextual information refinement.<n>AgentCPM-Explore achieves state-of-the-art (SOTA) performance among 4B-class models, matches or surpasses 8B-class SOTA models on four benchmarks, and even outperforms larger-scale models such as Claude-4.5-Sonnet or DeepSeek-v3.2 in five benchmarks.
  arXiv  Detail & Related papers  (2026-02-06T08:24:59Z)
• Augmenting Parameter-Efficient Pre-trained Language Models with Large Language Models [6.524460254566904]
  We introduce two strategies that use large language models to enhance the capabilities of pre-trained language models.<n>We empirically demonstrate that by combining parameter-efficient pre-trained models with large language models, we can improve the reliability and robustness of models.
  arXiv  Detail & Related papers  (2026-01-19T10:25:56Z)
• Youtu-LLM: Unlocking the Native Agentic Potential for Lightweight Large Language Models [78.73992315826035]
  We introduce Youtu-LLM, a lightweight language model that harmonizes high computational efficiency with native agentic intelligence.<n>Youtu-LLM is pre-trained from scratch to systematically cultivate reasoning and planning capabilities.
  arXiv  Detail & Related papers  (2025-12-31T04:25:11Z)
• The Application of Transformer-Based Models for Predicting Consequences of Cyber Attacks [0.4604003661048266]
  Threat Modeling can provide critical support to cybersecurity professionals, enabling them to take timely action and allocate resources that could be used elsewhere.<n>Recently, there has been a pressing need for automated methods to assess attack descriptions and forecast the future consequences of cyberattacks.<n>This study examines how Natural Language Processing (NLP) and deep learning can be applied to analyze the potential impact of cyberattacks.
  arXiv  Detail & Related papers  (2025-08-18T15:46:36Z)
• The Digital Cybersecurity Expert: How Far Have We Come? [49.89857422097055]
  We develop CSEBenchmark, a fine-grained cybersecurity evaluation framework based on 345 knowledge points expected of cybersecurity experts.<n>We evaluate 12 popular large language models (LLMs) on CSEBenchmark and find that even the best-performing model achieves only 85.42% overall accuracy.<n>By identifying and addressing specific knowledge gaps in each LLM, we achieve up to an 84% improvement in correcting previously incorrect predictions.
  arXiv  Detail & Related papers  (2025-04-16T05:36:28Z)
• CyberLLMInstruct: A Pseudo-malicious Dataset Revealing Safety-performance Trade-offs in Cyber Security LLM Fine-tuning [2.549390156222399]
  The integration of large language models into cyber security applications presents both opportunities and critical safety risks.<n>We introduce CyberLLMInstruct, a dataset of 54,928 pseudo-malicious instruction-response pairs spanning cyber security tasks.
  arXiv  Detail & Related papers  (2025-03-12T12:29:27Z)
• CTINexus: Automatic Cyber Threat Intelligence Knowledge Graph Construction Using Large Language Models [49.657358248788945]
  Textual descriptions in cyber threat intelligence (CTI) reports are rich sources of knowledge about cyber threats.<n>Current CTI knowledge extraction methods lack flexibility and generalizability.<n>We propose CTINexus, a novel framework for data-efficient CTI knowledge extraction and high-quality cybersecurity knowledge graph (CSKG) construction.
  arXiv  Detail & Related papers  (2024-10-28T14:18:32Z)
• Privacy Backdoors: Enhancing Membership Inference through Poisoning Pre-trained Models [112.48136829374741]
  In this paper, we unveil a new vulnerability: the privacy backdoor attack. When a victim fine-tunes a backdoored model, their training data will be leaked at a significantly higher rate than if they had fine-tuned a typical model. Our findings highlight a critical privacy concern within the machine learning community and call for a reevaluation of safety protocols in the use of open-source pre-trained models.
  arXiv  Detail & Related papers  (2024-04-01T16:50:54Z)
• SecureBERT and LLAMA 2 Empowered Control Area Network Intrusion Detection and Classification [2.824211356106516]
  We develop two distinct models for CAN intrusion detection: CAN-SecureBERT and CAN-LLAMA2. Can-LLAMA2 model surpasses the state-of-the-art models by achieving an exceptional performance 0.999993 in terms of balanced accuracy, precision detection rate, F1 score, and a remarkably low false alarm rate of 3.10e-6.
  arXiv  Detail & Related papers  (2023-11-19T23:49:08Z)
• Revolutionizing Cyber Threat Detection with Large Language Models: A privacy-preserving BERT-based Lightweight Model for IoT/IIoT Devices [3.340416780217405]
  This paper presents SecurityBERT, a novel architecture that leverages the Bidirectional Representations from Transformers (BERT) model for cyber threat detection in IoT networks. Our research demonstrates that SecurityBERT outperforms traditional Machine Learning (ML) and Deep Learning (DL) methods, such as Convolutional Neural Networks (CNNIoTs) or Recurrent Neural Networks (IoTRNNs) in cyber threat detection. SecurityBERT achieved an impressive 98.2% overall accuracy in identifying fourteen distinct attack types, surpassing previous records set by hybrid solutions.
  arXiv  Detail & Related papers  (2023-06-25T15:04:21Z)
• CySecBERT: A Domain-Adapted Language Model for the Cybersecurity Domain [0.0]
  We present a language model specifically tailored to the cybersecurity domain. The model is compared with other models based on 15 different domain-dependent extrinsic and intrinsic tasks. We show that our approach against catastrophic works, as the model is able to retrieve the previously trained domain-independent knowledge.
  arXiv  Detail & Related papers  (2022-12-06T13:49:12Z)
• Self-Destructing Models: Increasing the Costs of Harmful Dual Uses of Foundation Models [103.71308117592963]
  We present an algorithm for training self-destructing models leveraging techniques from meta-learning and adversarial learning. In a small-scale experiment, we show MLAC can largely prevent a BERT-style model from being re-purposed to perform gender identification.
  arXiv  Detail & Related papers  (2022-11-27T21:43:45Z)
• Plex: Towards Reliability using Pretrained Large Model Extensions [69.13326436826227]
  We develop ViT-Plex and T5-Plex, pretrained large model extensions for vision and language modalities, respectively. Plex greatly improves the state-of-the-art across reliability tasks, and simplifies the traditional protocol. We demonstrate scaling effects over model sizes up to 1B parameters and pretraining dataset sizes up to 4B examples.
  arXiv  Detail & Related papers  (2022-07-15T11:39:37Z)
• CyberLearning: Effectiveness Analysis of Machine Learning Security Modeling to Detect Cyber-Anomalies and Multi-Attacks [5.672898304129217]
  "CyberLearning" is a machine learning-based cybersecurity modeling with correlated-feature selection. We take into account binary classification model for detecting anomalies, and multi-class classification model for various types of cyber-attacks. We then present the artificial neural network-based security model considering multiple hidden layers.
  arXiv  Detail & Related papers  (2021-03-28T18:47:16Z)
• InfoBERT: Improving Robustness of Language Models from An Information Theoretic Perspective [84.78604733927887]
  Large-scale language models such as BERT have achieved state-of-the-art performance across a wide range of NLP tasks. Recent studies show that such BERT-based models are vulnerable facing the threats of textual adversarial attacks. We propose InfoBERT, a novel learning framework for robust fine-tuning of pre-trained language models.
  arXiv  Detail & Related papers  (2020-10-05T20:49:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.