Are Frontier LLMs Ready for Cybersecurity? Evidence for Vertical Foundation Models from Dual-Mode Vulnerability Benchmarks

• URL: http://arxiv.org/abs/2605.23243v1
• Date: Fri, 22 May 2026 05:24:43 GMT
• Title: Are Frontier LLMs Ready for Cybersecurity? Evidence for Vertical Foundation Models from Dual-Mode Vulnerability Benchmarks
• Abstract summary: We evaluate whether frontier LLMs are ready for cybersecurity through a dual-mode benchmark.<n>We test six frontier models (GPT-5.4, Codex5.3, Claude Opus4.6, Sonnet4.6, Gemini3.1Pro and Gemini3Flash) and two domain-specialized models across four testing paradigms.
• Score: 0.3303672705634661
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We evaluate whether frontier LLMs are ready for cybersecurity through a dual-mode benchmark: white-box function-level vulnerability detection (VulnLLM-R, across C/Java/Python) and black-box web application security testing (five production-style applications with 118 ground-truth vulnerabilities across 20+ CWE families, which we will open-source). We test six frontier models (GPT-5.4, Codex~5.3, Claude Opus~4.6, Sonnet~4.6, Gemini~3.1~Pro and Gemini~3~Flash) and two domain-specialized models across four testing paradigms. Our findings are sobering: (1)~every frontier model produces 10-50% false positive rates in white-box detection, systematically over-predicting vulnerabilities; (2)~in black-box testing, frontier models achieve only 4-8% ground-truth coverage, improving to just 10-19% even with external security tools (Playwright MCP, Burp Suite MCP); (3)~structured penetration-testing methodology encoded in domain-specialized agents raises per-family detection above 50%, demonstrating that methodology, not scale, is the primary lever; and (4)~a domain-specialized defense model achieves the highest precision (0.904) and lowest false positive rate (9.7%) among all models, on a single GPU. We identify the absence of structured security testing traces end-to-end request/response sequences, failure-heavy data, and multi-step attack chains as the fundamental training data bottleneck, and propose self-play security testing as a data generation strategy. Our results make the case for vertical foundation models purpose-built for cybersecurity.

Related papers

• Evaluating LLMs for Real-World Web Vulnerability Detection [0.0]
  Large Language Models (LLMs) have emerged as a promising tool for automated vulnerability detection.<n>This work benchmarks six frontier (Claude Opus 4.6, Codex GPT-5.4, Gemini 3.1-pro-preview) and open-weight models (Qwen 3.5, Qwen 3 Coder Next, MiniMax M2.5) on their ability to detect real-world web vulnerabilities.
  arXiv  Detail & Related papers  (2026-06-19T13:02:47Z)
• Comparative Analysis of Inference-Time Defense Methods for Multimodal Large Language Models [0.0]
  Multimodal large language models (MLLMs) now appear in safety-critical applications.<n>Retraining a model for each new vulnerability class is usually too expensive to be practical.<n>We report a comparative empirical evaluation of three-time defense methods and their combinations.
  arXiv  Detail & Related papers  (2026-06-09T14:13:54Z)
• Claw-Eval: Toward Trustworthy Evaluation of Autonomous Agents [66.97968363332465]
  We introduce Claw-Eval, an end-to-end evaluation suite addressing all three gaps in agent benchmarks.<n>Claw-Eval comprises 300 human-verified tasks spanning 9 categories across three groups.<n>Every agent action is recorded through three independent evidence channels.
  arXiv  Detail & Related papers  (2026-04-07T17:43:18Z)
• Your Agent, Their Asset: A Real-World Safety Analysis of OpenClaw [87.97230960702274]
  We present the first real-world safety evaluation of OpenClaw.<n>We introduce the CIK taxonomy, which unifies an agent's persistent state into three dimensions.<n>Our evaluations cover 12 attack scenarios on a live OpenClaw instance.
  arXiv  Detail & Related papers  (2026-04-06T15:27:05Z)
• CLASP: Defending Hybrid Large Language Models Against Hidden State Poisoning Attacks [48.54598003197356]
  State space models (SSMs) like Mamba have gained significant traction as efficient alternatives to Transformers.<n>HiSPAs, a recently discovered vulnerability that corrupts SSM memory through adversarial strings, pose a critical threat to these architectures.<n>We introduce the CLASP model to defend against this threat.
  arXiv  Detail & Related papers  (2026-03-12T17:29:55Z)
• Benchmarking LLAMA Model Security Against OWASP Top 10 For LLM Applications [0.0]
  This study benchmarks various Llama model variants against the Top 10 for LLM Applications framework.<n>We tested five standard Llama models and five Llama Guard variants on 100 adversarial prompts covering ten vulnerability categories.<n>The compact Llama-Guard-3-1B model achieved the highest detection rate of 76% with minimal latency.<n>We observe an inverse relationship between model size and security effectiveness, suggesting that smaller, specialized models often outperform larger general-purpose ones in security tasks.
  arXiv  Detail & Related papers  (2026-01-27T18:20:14Z)
• What Matters For Safety Alignment? [38.86339753409445]
  This paper presents a comprehensive empirical study on the safety alignment capabilities of AI systems.<n>We systematically investigate and compare the influence of six critical intrinsic model characteristics and three external attack techniques.<n>We identify the LRMs GPT-OSS-20B, Qwen3-Next-80B-A3B-Thinking, and GPT-OSS-120B as the top-three safest models.
  arXiv  Detail & Related papers  (2026-01-07T12:31:52Z)
• ALERT: Zero-shot LLM Jailbreak Detection via Internal Discrepancy Amplification [47.135407245022115]
  Existing detection methods mainly detect jailbreak status relying on jailbreak templates present in the training data.<n>We propose a layer-wise, module-wise, and token-wise amplification framework that progressively magnifies internal feature discrepancies between benign and jailbreak prompts.<n>Building upon these insights, we introduce ALERT, an efficient and effective zero-shot jailbreak detector.
  arXiv  Detail & Related papers  (2026-01-07T05:30:53Z)
• DiffuGuard: How Intrinsic Safety is Lost and Found in Diffusion Large Language Models [50.21378052667732]
  We conduct an in-depth analysis of dLLM vulnerabilities to jailbreak attacks across two distinct dimensions: intra-step and inter-step dynamics.<n>We propose DiffuGuard, a training-free defense framework that addresses vulnerabilities through a dual-stage approach.
  arXiv  Detail & Related papers  (2025-09-29T05:17:10Z)
• DyePack: Provably Flagging Test Set Contamination in LLMs Using Backdoors [52.85182605005619]
  We introduce DyePack, a framework that leverages backdoor attacks to identify models that used benchmark test sets during training.<n>Like how banks mix dye packs with their money to mark robbers, DyePack mixes backdoor samples with the test data to flag models that trained on it.<n>We evaluate DyePack on five models across three datasets, covering both multiple-choice and open-ended generation tasks.
  arXiv  Detail & Related papers  (2025-05-29T02:22:14Z)
• MOS: Towards Effective Smart Contract Vulnerability Detection through Mixture-of-Experts Tuning of Large Language Models [16.16186929130931]
  Smart contract vulnerabilities pose significant security risks to blockchain systems.<n>We propose a smart contract vulnerability detection framework based on mixture-of-experts tuning (MOE-Tuning) of large language models.<n> Experiments show that MOS significantly outperforms existing methods with average improvements of 6.32% in F1 score and 4.80% in accuracy.
  arXiv  Detail & Related papers  (2025-04-16T16:33:53Z)
• CyberLLMInstruct: A Pseudo-malicious Dataset Revealing Safety-performance Trade-offs in Cyber Security LLM Fine-tuning [2.549390156222399]
  The integration of large language models into cyber security applications presents both opportunities and critical safety risks.<n>We introduce CyberLLMInstruct, a dataset of 54,928 pseudo-malicious instruction-response pairs spanning cyber security tasks.
  arXiv  Detail & Related papers  (2025-03-12T12:29:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.