Structured but Fragile: On the Limits of LLMs in Cybersecurity Decision-Making

• URL: http://arxiv.org/abs/2608.20966v1
• Date: Fri, 21 Aug 2026 10:39:42 GMT
• Title: Structured but Fragile: On the Limits of LLMs in Cybersecurity Decision-Making
• Abstract summary: Large language models (LLMs) are increasingly used in cybersecurity, yet it remains unclear whether they can perform structured security reasoning.<n>We study this question in the context of defence selection over attack graphs derived from real-world threat scenarios.<n>Our findings show that LLMs can approximate structured cybersecurity reasoning under controlled representations, but do not apply it robustly.
• Score: 2.4336422314992916
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models (LLMs) are increasingly used in cybersecurity workflows, yet it remains unclear whether they can perform structured security reasoning or merely rely on superficial cues and prior knowledge. We study this question in the context of defence selection over attack graphs derived from real-world threat scenarios, including ransomware, supply-chain compromise, cloud abuse, Kubernetes attacks, POS malware, and ICS/OT intrusion. Given a budget constraint, LLMs must select security controls to minimise attacker success. We compare their strategies against each other and against a game-theoretic optimization baseline used as a normative reference for structured reasoning. Our results show that LLMs exhibit conditional competence. When explicit attack-graph structure is provided, they often produce coherent strategies close to the optimization baseline. However, their capabilities are fragile. LLM behaviour becomes increasingly fragile with graph complexity and is highly sensitive to framing. Small prompt changes can substantially alter rankings, and merely relabeling a poor strategy as ``optimal'' dramatically improves its evaluation. We further observe a non-monotonic relationship between formal risk and LLM judgement: strategies closest to the optimum are not necessarily ranked highest by LLM evaluators. To further probe reasoning ability, we ask LLMs to generate solvers for the same optimization problem. While the generated implementations recover the correct high-level formulation, they scale poorly compared to a purpose-built solver. Overall, our findings show that LLMs can approximate structured cybersecurity reasoning under controlled representations, but do not apply it robustly. This has important implications for the design and evaluation of AI-assisted security decision-support systems.

Related papers

• METER: Evaluating Multi-Level Contextual Causal Reasoning in Large Language Models [61.33372454250959]
  Contextual causal reasoning is a critical yet challenging capability for Large Language Models.<n>Existing benchmarks fail to ensure context consistency or cover the full causal hierarchy.<n>We pioneer METER to systematically benchmark LLMs across all three levels of the causal ladder.
  arXiv  Detail & Related papers  (2026-04-13T14:07:11Z)
• The Vulnerability of LLM Rankers to Prompt Injection Attacks [40.03039307576983]
  Large Language Models (LLMs) have emerged as powerful re-rankers.<n>Recent research has showed that simple prompt injections embedded within a candidate document can significantly alter an LLM's ranking decisions.
  arXiv  Detail & Related papers  (2026-02-18T06:19:08Z)
• PSM: Prompt Sensitivity Minimization via LLM-Guided Black-Box Optimization [0.0]
  This paper introduces a novel framework for hardening system prompts through shield appending.<n>We leverage an LLM-as-optimizer to search the space of possible SHIELDs, seeking to minimize a leakage metric derived from a suite of adversarial attacks.<n>We demonstrate empirically that our optimized SHIELDs significantly reduce prompt leakage against a comprehensive set of extraction attacks.
  arXiv  Detail & Related papers  (2025-11-20T10:25:45Z)
• Reasoning with Confidence: Efficient Verification of LLM Reasoning Steps via Uncertainty Heads [104.9566359759396]
  We propose a lightweight alternative for step-level reasoning verification based on data-driven uncertainty scores.<n>Our findings suggest that the internal states of LLMs encode their uncertainty and can serve as reliable signals for reasoning verification.
  arXiv  Detail & Related papers  (2025-11-09T03:38:29Z)
• Towards Reliable and Practical LLM Security Evaluations via Bayesian Modelling [1.0266286487433585]
  It is critical to understand vulnerabilities accurately before adopting a new large language model (LLM) architecture.<n>Existing evaluations can be difficult to trust, often drawing conclusions from LLMs that are not meaningfully comparable.<n>We propose a principled and practical end-to-end framework for evaluating LLM vulnerabilities to prompt injection attacks.
  arXiv  Detail & Related papers  (2025-10-07T09:22:22Z)
• Can LLMs effectively provide game-theoretic-based scenarios for cybersecurity? [51.96049148869987]
  Large Language Models (LLMs) offer new tools and challenges for the security of computer systems.<n>We investigate whether classical game-theoretic frameworks can effectively capture the behaviours of LLM-driven actors and bots.
  arXiv  Detail & Related papers  (2025-08-04T08:57:14Z)
• LogicGuard: Improving Embodied LLM agents through Temporal Logic based Critics [3.3890411643175646]
  Large language models (LLMs) have shown promise in zero-shot and single step reasoning and decision making problems.<n>We introduce LogicGuard, a modular actor-critic architecture in which an LLM actor is guided by a trajectory level LLM critic.<n>Our setup combines the reasoning strengths of language models with the guarantees of formal logic.
  arXiv  Detail & Related papers  (2025-07-04T04:53:53Z)
• AegisLLM: Scaling Agentic Systems for Self-Reflective Defense in LLM Security [74.22452069013289]
  AegisLLM is a cooperative multi-agent defense against adversarial attacks and information leakage.<n>We show that scaling agentic reasoning system at test-time substantially enhances robustness without compromising model utility.<n> Comprehensive evaluations across key threat scenarios, including unlearning and jailbreaking, demonstrate the effectiveness of AegisLLM.
  arXiv  Detail & Related papers  (2025-04-29T17:36:05Z)
• Adversarial Reasoning at Jailbreaking Time [49.70772424278124]
  Large language models (LLMs) are becoming more capable and widespread.<n>Recent advances in standardizing, measuring, and scaling test-time compute suggest new methodologies for optimizing models to achieve high performance on hard tasks.<n>In this paper, we apply these advances to the task of model jailbreaking: eliciting harmful responses from aligned LLMs.
  arXiv  Detail & Related papers  (2025-02-03T18:59:01Z)
• Targeting the Core: A Simple and Effective Method to Attack RAG-based Agents via Direct LLM Manipulation [4.241100280846233]
  AI agents, powered by large language models (LLMs), have transformed human-computer interactions by enabling seamless, natural, and context-aware communication.<n>This paper investigates a critical vulnerability: adversarial attacks targeting the LLM core within AI agents.
  arXiv  Detail & Related papers  (2024-12-05T18:38:30Z)
• SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Models [107.82336341926134]
  SALAD-Bench is a safety benchmark specifically designed for evaluating Large Language Models (LLMs) It transcends conventional benchmarks through its large scale, rich diversity, intricate taxonomy spanning three levels, and versatile functionalities.
  arXiv  Detail & Related papers  (2024-02-07T17:33:54Z)
• Advancing TTP Analysis: Harnessing the Power of Large Language Models with Retrieval Augmented Generation [1.2289361708127877]
  It is unclear how Large Language Models (LLMs) can be used in an efficient and proper way to provide accurate responses for critical domains such as cybersecurity. This work studies and compares the uses of supervised fine-tuning (SFT) of encoder-only LLMs vs. Retrieval Augmented Generation (RAG) for decoder-only LLMs. Our studies show decoder-only LLMs with RAG achieves better performance than encoder-only models with SFT.
  arXiv  Detail & Related papers  (2023-12-30T16:56:24Z)
• Baseline Defenses for Adversarial Attacks Against Aligned Language Models [109.75753454188705]
  Recent work shows that text moderations can produce jailbreaking prompts that bypass defenses. We look at three types of defenses: detection (perplexity based), input preprocessing (paraphrase and retokenization), and adversarial training. We find that the weakness of existing discretes for text, combined with the relatively high costs of optimization, makes standard adaptive attacks more challenging for LLMs.
  arXiv  Detail & Related papers  (2023-09-01T17:59:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.