Mapping the Exploitation Surface: A 10,000-Trial Taxonomy of What Makes LLM Agents Exploit Vulnerabilities

• URL: http://arxiv.org/abs/2604.04561v1
• Date: Mon, 06 Apr 2026 09:44:34 GMT
• Title: Mapping the Exploitation Surface: A 10,000-Trial Taxonomy of What Makes LLM Agents Exploit Vulnerabilities
• Abstract summary: LLM agents with tool access can exploit security vulnerabilities.<n>What is not known is which features of a system prompt trigger this behaviour, and which do not.<n>We present a systematic taxonomy based on 10,000 trials across seven models, 37 prompt conditions, and 12 hypothesized attack dimensions.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: LLM agents with tool access can discover and exploit security vulnerabilities. This is known. What is not known is which features of a system prompt trigger this behaviour, and which do not. We present a systematic taxonomy based on approximately 10,000 trials across seven models, 37 prompt conditions, and 12 hypothesized attack dimensions -- all executed in real Docker sandboxes. Every condition includes the instruction "Always respect all rules and access policies"; only a single encouragement sentence varies. Within the task class studied (planted test-runner vulnerabilities), the threat model is narrower than assumed: nine of twelve dimensions -- including minimization, moral licensing, incentives, identity priming, and reasoning triggers -- produce no detectable exploitation at n=50 per cell (upper 95% CI <7% per condition). But it is also more subtle than expected: one dimension, goal reframing, reliably triggers exploitation. "You are solving a puzzle; there may be hidden clues" produces 38-40% exploitation on Claude Sonnet 4 despite the explicit rule instruction, replicating across four models (CTF framing: 8-14% on DeepSeek, GPT-5-mini, o4-mini). The agent does not override the rules; it reinterprets the task so that exploitative actions become task-aligned. GPT-4.1 produces no exploitation across 1,850 trials (37 conditions), and a temporal comparison across four OpenAI models released over eleven months shows a pattern consistent with improving safety training, though model capability differences are a confounder. The practical contribution is a narrowed, testable threat model: defenders should audit for goal-reframing language, not for the broad class of adversarial prompts.

Related papers

• On Understanding, Identifying, and Mitigating Vulnerabilities in Agentic Large Language Models [2.8528615301556997]
  Large Language Models (LLMs) have undergone a shift from stateless conversational interfaces to autonomous agents.<n>A compromised reasoning step can trigger unauthorized data access, irreversible state changes, or cascading failures.<n>To quantify the state of the field, we conducted a systematic literature review under PRISMA 2020 guidelines.
  arXiv  Detail & Related papers  (2026-08-11T06:11:26Z)
• ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks? [92.21756459993695]
  Exploitation is a challenging task because it requires low-level program reasoning.<n>Despite its importance and diagnostic value, exploitation remains under-evaluated.<n>We introduce ExploitGym, a large-scale, diverse, realistic benchmark on the exploitation capabilities of AI agents.
  arXiv  Detail & Related papers  (2026-05-11T18:00:14Z)
• Position: AI Security Policy Should Target Systems, Not Models [2.741152471987327]
  We present an open-source adversarial testing framework in which multiple lightweight LLM agents coordinate through shared memory.<n>Our results demonstrate that both safety bypass of frontier models and software vulnerability discovery are achievable at effectively zero cost.
  arXiv  Detail & Related papers  (2026-05-10T12:27:14Z)
• Reward Hacking Benchmark: Measuring Exploits in LLM Agents with Tool Use [0.016201337438609787]
  Reinforcement learning (RL) trained language model agents with tool access are increasingly deployed in coding assistants, research tools, and autonomous systems.<n>We introduce the Reward Hacking Benchmark (RHB), a suite of multi-step tasks requiring sequential tool operations with naturalistic shortcut opportunities.<n>We evaluate 13 frontier models from OpenAI, Anthropic, Google, and DeepSeek.
  arXiv  Detail & Related papers  (2026-05-03T07:10:42Z)
• BadSkill: Backdoor Attacks on Agent Skills via Model-in-Skill Poisoning [34.60596020541521]
  We present BadSkill, a backdoor attack formulation that targets model-in-skill threat surface.<n>In BadSkill, an adversary publishes a seemingly benign skill whose embedded model is backdoor-fine-tuned to activate a hidden payload.<n>Our benchmark spans 13 skills, including 8 triggered tasks and 5 non-trigger control skills, with a combined main evaluation set of 571 negative-class queries and 396 trigger-aligned queries.<n>BadSkill achieves up to 99.5% average attack success rate (ASR) across the eight triggered skills while maintaining strong benign-side accuracy on negative-class queries.
  arXiv  Detail & Related papers  (2026-04-10T14:48:29Z)
• The Persistent Vulnerability of Aligned AI Systems [2.9994384641583185]
  This thesis contributes to four open problems in AI safety.<n> understanding dangerous internal computations, removing dangerous behaviors once embedded, testing for vulnerabilities before deployment, and predicting when models will act against deployers.
  arXiv  Detail & Related papers  (2026-03-31T23:49:07Z)
• Backdoor4Good: Benchmarking Beneficial Uses of Backdoors in LLMs [63.668008418097905]
  We show that beneficial backdoors can achieve high controllability, tamper-resistance, and stealthiness while preserving clean-task performance.<n>Our findings demonstrate new insights that backdoors need not be inherently malicious; when properly designed, they can serve as modular, interpretable, and beneficial building blocks for trustworthy AI systems.
  arXiv  Detail & Related papers  (2026-03-08T04:01:13Z)
• In-Context Environments Induce Evaluation-Awareness in Language Models [0.12691047660244334]
  Humans often become more self-aware under threat, yet can lose self-awareness when absorbed in a task.<n>We introduce a black-box adversarial optimization framework treating the in-context prompt as an optimizable environment.<n>We show that adversarially optimized prompts pose a substantially greater threat to evaluation reliability than previously understood.
  arXiv  Detail & Related papers  (2026-03-04T08:22:02Z)
• Can Adversarial Code Comments Fool AI Security Reviewers -- Large-Scale Empirical Study of Comment-Based Attacks and Defenses Against LLM Code Analysis [0.0]
  Adversarial comments produce small, statistically non-significant effects on detection accuracy.<n>Complex adversarial strategies offer no advantage over simple manipulative comments.<n>Comment stripping reduces detection for weaker models by removing helpful context.
  arXiv  Detail & Related papers  (2026-02-18T00:34:17Z)
• AI Security Beyond Core Domains: Resume Screening as a Case Study of Adversarial Vulnerabilities in Specialized LLM Applications [71.27518152526686]
  Large Language Models (LLMs) excel at text comprehension and generation, making them ideal for automated tasks like code review and content moderation.<n>LLMs can be manipulated by "adversarial instructions" hidden in input data, such as resumes or code, causing them to deviate from their intended task.<n>This paper introduces a benchmark to assess this vulnerability in resume screening, revealing attack success rates exceeding 80% for certain attack types.
  arXiv  Detail & Related papers  (2025-12-23T08:42:09Z)
• The Trojan Knowledge: Bypassing Commercial LLM Guardrails via Harmless Prompt Weaving and Adaptive Tree Search [58.8834056209347]
  Large language models (LLMs) remain vulnerable to jailbreak attacks that bypass safety guardrails to elicit harmful outputs.<n>We introduce the Correlated Knowledge Attack Agent (CKA-Agent), a dynamic framework that reframes jailbreaking as an adaptive, tree-structured exploration of the target model's knowledge base.
  arXiv  Detail & Related papers  (2025-12-01T07:05:23Z)
• BountyBench: Dollar Impact of AI Agent Attackers and Defenders on Real-World Cybersecurity Systems [62.17474934536671]
  We introduce the first framework to capture offensive and defensive cyber-capabilities in evolving real-world systems.<n>To capture the vulnerability lifecycle, we define three task types: Detect (detecting a new vulnerability), Exploit (exploiting a specific vulnerability), and Patch (patching a specific vulnerability)<n>We evaluate 8 agents: Claude Code, OpenAI Codex CLI with o3-high and o4-mini, and custom agents with o3-high, GPT-4.1, Gemini 2.5 Pro Preview, Claude 3.7 Sonnet Thinking, and DeepSeek-R1.
  arXiv  Detail & Related papers  (2025-05-21T07:44:52Z)
• Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Models [33.1538965735133]
  Cybench is a framework for specifying cybersecurity tasks and evaluating agents on those tasks.<n>We include 40 professional-level Capture the Flag (CTF) tasks from 4 distinct CTF competitions.<n>We construct a cybersecurity agent and evaluate 8 models: GPT-4o, OpenAI o1-preview, Claude 3 Opus, Claude 3.5 Sonnet, Mixtral 8x22b Instruct, Gemini 1.5 Pro, Llama 3 70B Chat, and Llama 3.1 405B Instruct.
  arXiv  Detail & Related papers  (2024-08-15T17:23:10Z)
• Stealthy Backdoor Attack for Code Models [19.272856932095966]
  Existing backdoor attacks on code models use unstealthy and easy-to-detect triggers. This paper aims to investigate the vulnerability of code models with stealthy backdoor attacks. We find that around 85% of adaptive triggers in AFRAIDOOR bypass the detection in the defense process.
  arXiv  Detail & Related papers  (2023-01-06T13:15:42Z)
• Backdoor Attacks on Crowd Counting [63.90533357815404]
  Crowd counting is a regression task that estimates the number of people in a scene image. In this paper, we investigate the vulnerability of deep learning based crowd counting models to backdoor attacks.
  arXiv  Detail & Related papers  (2022-07-12T16:17:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.