Threat Modelling using Domain-Adapted Language Models: Empirical Evaluation and Insights

• URL: http://arxiv.org/abs/2605.10808v1
• Date: Mon, 11 May 2026 16:31:25 GMT
• Title: Threat Modelling using Domain-Adapted Language Models: Empirical Evaluation and Insights
• Abstract summary: Large Language Models (LLMs) are increasingly explored for cybersecurity applications such as vulnerability detection.<n>In this study, we extend the research area of structured threat modelling by systematically evaluating domain-adapted language models.<n>We use both LLMs and Small Language Models(SLMs) that were domain adapted to telecommunications and cybersecuirty.
• Score: 0.45880283710344066
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large Language Models(LLMs) are increasingly explored for cybersecurity applications such as vulnerability detection. In the domain of threat modelling, prior work has primarily evaluated a number of general-purpose Large Language Models under limited prompting settings. In this study, we extend the research area of structured threat modelling by systematically evaluating domain-adapted language models of different sizes to their general counterparts. We use both LLMs and Small Language Models(SLMs) that were domain adapted to telecommunications and cybersecuirty. For the structured threat modelling, we selected the widely used STRIDE approach and the application area is 5G security. We present a comprehensive empirical evaluation using 52 different configurations (on 8 different language models) to analyze the impact of 1) domain adaptation, 2) model scale, 3) decoding strategies (greedy vs. stochastic sampling), and 4) prompting technique on STRIDE threat classification. Our results show that domain-adapted models do not consistently outperform their general-purpose counterparts, and decoding strategies significantly affect model behavior and output validity. They also show that while larger models generally achieve higher performance, these gains are neither consistent nor sufficient for reliable threat modelling. These findings highlight fundamental limitations of current LLMs for structured threat modelling tasks and suggest that improvements require more than additional training data or model scaling, motivating the need for incorporating more task-specific reasoning and stronger grounding in security concepts. We present insights on invalid outputs encountered and present suggestions for prompting tailored specifically for STRIDE threat modelling.

Related papers

• Augmenting Parameter-Efficient Pre-trained Language Models with Large Language Models [6.524460254566904]
  We introduce two strategies that use large language models to enhance the capabilities of pre-trained language models.<n>We empirically demonstrate that by combining parameter-efficient pre-trained models with large language models, we can improve the reliability and robustness of models.
  arXiv  Detail & Related papers  (2026-01-19T10:25:56Z)
• Bayesian Low-Rank Factorization for Robust Model Adaptation [60.296574524609575]
  We explore factorized adapters for speech foundation models, which place priors near zero to achieve sparser adaptation matrices.<n>Our results show only minimal adaptation loss while significantly reducing catastrophic forgetting of the base model.
  arXiv  Detail & Related papers  (2025-10-21T15:23:30Z)
• Tiny Reward Models [0.0]
  We present TinyRM, a family of small, bidirectional masked language models (MLMs) with as few as 400 million parameters.<n>TinyRM combines FLAN-style prompting, Directional Low-Rank Adaptation (DoRA), and layer freezing to achieve strong performance on RewardBench.<n>Our experiments suggest that small models benefit from domain-specific tuning strategies, particularly in reasoning.
  arXiv  Detail & Related papers  (2025-07-14T06:43:00Z)
• Graph Representation-based Model Poisoning on Federated Large Language Models [3.5233863453805143]
  Federated large language models (FedLLMs) enable powerful generative capabilities within wireless networks while preserving data privacy.<n>This article first reviews recent advancements in model poisoning techniques and existing defense mechanisms for FedLLMs, underscoring critical limitations.<n>The article further investigates graph representation-based model poisoning (GRMP), an emerging attack paradigm that exploits higher-order correlations among benign client gradients to craft malicious updates indistinguishable from legitimate ones.
  arXiv  Detail & Related papers  (2025-07-02T13:20:52Z)
• A Survey on Model Extraction Attacks and Defenses for Large Language Models [55.60375624503877]
  Model extraction attacks pose significant security threats to deployed language models.<n>This survey provides a comprehensive taxonomy of extraction attacks and defenses, categorizing attacks into functionality extraction, training data extraction, and prompt-targeted attacks.<n>We examine defense mechanisms organized into model protection, data privacy protection, and prompt-targeted strategies, evaluating their effectiveness across different deployment scenarios.
  arXiv  Detail & Related papers  (2025-06-26T22:02:01Z)
• The CoT Encyclopedia: Analyzing, Predicting, and Controlling how a Reasoning Model will Think [81.38614558541772]
  We introduce the CoT Encyclopedia, a framework for analyzing and steering model reasoning.<n>Our method automatically extracts diverse reasoning criteria from model-generated CoTs.<n>We show that this framework produces more interpretable and comprehensive analyses than existing methods.
  arXiv  Detail & Related papers  (2025-05-15T11:31:02Z)
• Membership Inference Attacks on Large-Scale Models: A Survey [5.795582095405318]
  Membership Inference Attacks (MIAs) are an important technique for exposing or assessing privacy risks.<n>We provide the first comprehensive review of MIAs targeting Large Language Models (LLMs) and Large Multimodal Models (LMMs)<n>Unlike prior surveys, we further examine MIAs across multiple stages of the model pipeline, including pre-training, fine-tuning, alignment, and Retrieval-Augmented Generation (RAG)
  arXiv  Detail & Related papers  (2025-03-25T04:11:47Z)
• Privacy Backdoors: Enhancing Membership Inference through Poisoning Pre-trained Models [112.48136829374741]
  In this paper, we unveil a new vulnerability: the privacy backdoor attack. When a victim fine-tunes a backdoored model, their training data will be leaked at a significantly higher rate than if they had fine-tuned a typical model. Our findings highlight a critical privacy concern within the machine learning community and call for a reevaluation of safety protocols in the use of open-source pre-trained models.
  arXiv  Detail & Related papers  (2024-04-01T16:50:54Z)
• Evaluating Concurrent Robustness of Language Models Across Diverse Challenge Sets [46.19529338280716]
  Language models, characterized by their black-box nature, often hallucinate and display sensitivity to input perturbations.<n>We introduce a methodology designed to examine how input perturbations affect language models across various scales.<n>We present three distinct fine-tuning strategies to address robustness against multiple perturbations.
  arXiv  Detail & Related papers  (2023-11-15T02:59:10Z)
• SecurityNet: Assessing Machine Learning Vulnerabilities on Public Models [74.58014281829946]
  We analyze the effectiveness of several representative attacks/defenses, including model stealing attacks, membership inference attacks, and backdoor detection on public models. Our evaluation empirically shows the performance of these attacks/defenses can vary significantly on public models compared to self-trained models.
  arXiv  Detail & Related papers  (2023-10-19T11:49:22Z)
• CySecBERT: A Domain-Adapted Language Model for the Cybersecurity Domain [0.0]
  We present a language model specifically tailored to the cybersecurity domain. The model is compared with other models based on 15 different domain-dependent extrinsic and intrinsic tasks. We show that our approach against catastrophic works, as the model is able to retrieve the previously trained domain-independent knowledge.
  arXiv  Detail & Related papers  (2022-12-06T13:49:12Z)
• Large Language Models with Controllable Working Memory [64.71038763708161]
  Large language models (LLMs) have led to a series of breakthroughs in natural language processing (NLP) What further sets these models apart is the massive amounts of world knowledge they internalize during pretraining. How the model's world knowledge interacts with the factual information presented in the context remains under explored.
  arXiv  Detail & Related papers  (2022-11-09T18:58:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.