How Reliable Are AI Attackers Against a Fixed Vulnerable Target? A 400-Run Empirical Study of LLM Penetration Testing Consistency

• URL: http://arxiv.org/abs/2605.30096v1
• Date: Thu, 28 May 2026 15:39:43 GMT
• Title: How Reliable Are AI Attackers Against a Fixed Vulnerable Target? A 400-Run Empirical Study of LLM Penetration Testing Consistency
• Abstract summary: Large language models (LLMs) can autonomously conduct multi-stage cyber attacks, but the consistency of their offensive behavior under repeated penetration unstudied.<n>This work presents the first large-scale empirical measurement of attack consistency: 400 runs (4 models, 100 each) against an identical honeypot hosting Juice Shop.<n>No model emitted a content refusal that survived the orchestrator's one-shot authorization re-prompt at 0-1.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models (LLMs) can autonomously conduct multi-stage cyber attacks, but the consistency of their offensive behavior under repeated trials remains unstudied. This work presents the first large-scale empirical measurement of LLM attack consistency: 400 autonomous penetration testing runs (4 models, 100 each) against an identical honeypot hosting OWASP Juice Shop and two additional vulnerable services, holding prompt, orchestrator, and target constant. No model emitted a content refusal that survived the orchestrator's one-shot authorization re-prompt at iterations 0-1. Claude Sonnet 4's API calls did encounter upstream service unavailability - 91 of 1,135 calls returned HTTP 529 overloaded_error during a documented Anthropic capacity event, truncating 39 of 100 Claude runs. An earlier draft catalogued these as safety refusals; on full-log audit they are upstream API failures, not model-level refusals. Despite this, Claude achieved full exploitation in 61 of 100 runs; Gemini 2.5 Flash-Lite in 85; GPT-4o-mini in 56 while deploying 98 unique attack strategies; qwen2.5-coder:14b in 25. Failure modes are model-distinctive: Claude through API truncation (39 runs), qwen through premature completion (52), GPT-4o-mini through iteration-budget exhaustion (23). Cross-service credential reuse appeared only in configurations retaining the most conversation history (qwen 57%, GPT-4o-mini 49%, cloud models 0% on 5-exchange windows). Cross-model exploitation rate differences are statistically significant (p < 0.001) with large effect sizes; qwen vs. Gemini SQL injection rates differ at Cohen's h = 1.12. First-exploit timing fell within a 15-30 second wall-clock range. To our knowledge, this is the first study to measure autonomous LLM attack behavior at N=100 per model across a multi-service target.

Related papers

• FuzzingBrain-Bench V1: Evaluating Open-Ended Bug Discovery by LLMs [46.58695049126903]
  We present FuzzingBrain-Bench, a benchmark for assessing AI models' ability to discover bugs in open-source software.<n>A model's performance on each challenge is scored based on the number of distinct crash signatures it produces.<n>Claude Opus 4.8 performs best, triggering crashes in 60 of 77 challenges and achieving a score of 196 out of 579.
  arXiv  Detail & Related papers  (2026-08-25T21:09:20Z)
• ClawArena-Team: Benchmarking Subagent Orchestration and Dynamic Workflows in Language-Model Agents [61.184954205350984]
  ClawArena-Team is a benchmark of 41 multi-turn, multimodal, multi-agent scenarios spanning 258 evaluation rounds and 72 staged updates.<n>All scoring is execution-based with no LLM judge: an overall score -- the Subagent-Management Score (SMS) -- multiplies task correctness by a least-privilege and modality-routing factor.
  arXiv  Detail & Related papers  (2026-06-30T06:06:08Z)
• Understanding and Evaluating Claw-like Agent Security Through a Computer-Systems Lens [65.53494487819053]
  Claw-like AI agents (e.g., OpenClaw) are always-on processes with persistent access to credentials, files, tools, and external services.<n>We adopt a computer-system analogy: treating a Claw-like agent as an agentic computer system whose gateway runtime plays an OS-like mediation role.<n>We develop SafeClawArena, a benchmark of 406 adversarial tasks across four attack surfaces.
  arXiv  Detail & Related papers  (2026-06-29T18:00:45Z)
• Overeager Coding Agents: Measuring Out-of-Scope Actions on Benign Tasks [40.270213696031625]
  OverEager-Gen is a benchmark dedicated to overeager behavior on benign tasks.<n>On Claude Code, stripping the consent declaration alone raises the overeager rate from 0.0% to 17.1%.<n>OverEager-Gen therefore certifies each scenario's discriminative power before admission.
  arXiv  Detail & Related papers  (2026-05-18T16:00:41Z)
• The Range Shrinks, the Threat Remains: Re-evaluating LLM Package Hallucinations on the 2026 Frontier-Model Cohort [51.56484100374058]
  Spracklen et al. showed that code-generating large language models hallucinate package names that do not exist on PyPI or npm.<n>We measure hallucination rates across 199,845 paired Python and JavaScript prompts validated against PyPI and npm master lists.<n>We identify a set of 127 package names (109 on PyPI, 18 on npm) that all five evaluated models invent identically.
  arXiv  Detail & Related papers  (2026-05-16T16:08:52Z)
• Oracle Poisoning: Corrupting Knowledge Graphs to Weaponise AI Agent Reasoning [0.9236074230806578]
  Oracle Poisoning is an attack class in which an adversary corrupts a structured knowledge graph.<n>Unlike prompt injection, Oracle Poisoning manipulates the data agents reason over, not their instructions.<n>We demonstrate six attack scenarios against a production 42-million-node code knowledge graph.
  arXiv  Detail & Related papers  (2026-05-10T23:55:13Z)
• Position: AI Security Policy Should Target Systems, Not Models [2.741152471987327]
  We present an open-source adversarial testing framework in which multiple lightweight LLM agents coordinate through shared memory.<n>Our results demonstrate that both safety bypass of frontier models and software vulnerability discovery are achievable at effectively zero cost.
  arXiv  Detail & Related papers  (2026-05-10T12:27:14Z)
• MOSAIC-Bench: Measuring Compositional Vulnerability Induction in Coding Agents [2.1942030377331245]
  Coding agents often pass per-prompt safety review yet ship exploitable code when their tasks are decomposed into routine engineering tickets.<n>We introduce MOSAIC-Bench, a benchmark of 199 three-stage attack chains paired with deterministic exploit oracles on deployed software substrates.<n>We show that nine production coding agents compose innocuous tickets at 53-86% end-to-end ASR with only two refusals across all staged runs.
  arXiv  Detail & Related papers  (2026-05-05T16:38:23Z)
• Your Agent Is Mine: Measuring Malicious Intermediary Attacks on the LLM Supply Chain [6.142057368768942]
  Large language model (LLM) agents increasingly rely on third-party API routers to dispatch tool-calling requests.<n>We present the first systematic study of this attack surface.
  arXiv  Detail & Related papers  (2026-04-09T16:06:41Z)
• How Vulnerable Are AI Agents to Indirect Prompt Injections? Insights from a Large-Scale Public Competition [48.32744727426218]
  LLM based agents are increasingly deployed in high stakes settings where they process external data sources such as emails, documents, and code repositories.<n>This creates exposure to indirect prompt injection attacks, where adversarial instructions embedded in external content manipulate agent behavior without user awareness.<n>We present findings from a large scale public red teaming competition evaluating this dual objective across three agent settings.
  arXiv  Detail & Related papers  (2026-03-16T14:49:36Z)
• AI Security Beyond Core Domains: Resume Screening as a Case Study of Adversarial Vulnerabilities in Specialized LLM Applications [71.27518152526686]
  Large Language Models (LLMs) excel at text comprehension and generation, making them ideal for automated tasks like code review and content moderation.<n>LLMs can be manipulated by "adversarial instructions" hidden in input data, such as resumes or code, causing them to deviate from their intended task.<n>This paper introduces a benchmark to assess this vulnerability in resume screening, revealing attack success rates exceeding 80% for certain attack types.
  arXiv  Detail & Related papers  (2025-12-23T08:42:09Z)
• Revisiting Backdoor Attacks on LLMs: A Stealthy and Practical Poisoning Framework via Harmless Inputs [54.90315421117162]
  We propose a novel poisoning method via completely harmless data.<n>Inspired by the causal reasoning in auto-regressive LLMs, we aim to establish robust associations between triggers and an affirmative response prefix.<n>We observe an interesting resistance phenomenon where the LLM initially appears to agree but subsequently refuses to answer.
  arXiv  Detail & Related papers  (2025-05-23T08:13:59Z)
• AegisLLM: Scaling Agentic Systems for Self-Reflective Defense in LLM Security [74.22452069013289]
  AegisLLM is a cooperative multi-agent defense against adversarial attacks and information leakage.<n>We show that scaling agentic reasoning system at test-time substantially enhances robustness without compromising model utility.<n> Comprehensive evaluations across key threat scenarios, including unlearning and jailbreaking, demonstrate the effectiveness of AegisLLM.
  arXiv  Detail & Related papers  (2025-04-29T17:36:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.