A New Framework for Cybersecurity Refusals in AI Agents

• URL: http://arxiv.org/abs/2606.02644v1
• Date: Sun, 31 May 2026 15:39:39 GMT
• Title: A New Framework for Cybersecurity Refusals in AI Agents
• Abstract summary: We present the first framework for establishing refusal boundaries in offensive security contexts.<n>We apply this framework to assess how current LLM-powered agents adhere to appropriate refusal boundaries across a range of web-based offensive security scenarios.
• Score: 52.94784168139071
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Agentic scaffolds have dramatically improved LLM performance on complex, long-horizon tasks, yielding both broad benefits and amplified risks in domains like cybersecurity. Existing benchmarks for AI agents in cybersecurity focus mainly on measuring proficiency--how effectively agents can complete offensive security tasks--but neglect a critical question: when and how should agents refuse harmful requests? We present the first framework for establishing refusal boundaries in offensive security contexts. Our framework defines (1) principled criteria for when tasks should be refused, (2) categories of tasks that warrant refusal, and (3) evaluation methodology for measuring agent robustness under both benign and adversarial conditions. We apply this framework to assess how current LLM-powered agents adhere to appropriate refusal boundaries across a range of web-based offensive security scenarios, finding that 6 of 8 frontier models tested show near-zero refusal rates, with only 2 models (GPT-5.2 and GPT-5.1 Codex) demonstrating any meaningful refusal behavior.

Related papers

• SecRespond: Benchmarking AI Agents for Real-World Post-Compromise Incident Response [38.89333299127352]
  We introduce SecRespond, the first benchmark for evaluating Large Language Model (LLM) agents on the post-compromise incident-response workflow.<n>We instantiate this task across 10 cyber ranges, each constructed from a distinct compromised cloud host.<n> Experimental results show that although current agents can reliably uncover the problems exposed by alerts, they struggle to proactively investigate the disk for silent intrusions.
  arXiv  Detail & Related papers  (2026-07-29T11:32:23Z)
• The Containment Gap: How Deployed Agentic AI Frameworks Fail Public-Facing Safety Requirements [4.431419229831417]
  Agentic large language model systems are increasingly deployed in public-facing domains.<n>We ask whether the frameworks used to build these systems provide architectural-level structural safety guarantees.
  arXiv  Detail & Related papers  (2026-06-11T01:46:26Z)
• AgentCanary: A Security Evaluation Framework for Autonomous AI Agents in Real Executable Environments [32.982317538632806]
  We propose AgentCanary, a comprehensive security evaluation framework for autonomous AI agents.<n>AgentCanary provides a systematic solution along three contributions.<n>We evaluate a broad set of frontier models on AgentCanary against multiple established adversarial attack methods.
  arXiv  Detail & Related papers  (2026-06-09T06:55:37Z)
• Architecting Secure AI Agents: Perspectives on System-Level Defenses Against Indirect Prompt Injection Attacks [46.32011684548212]
  AI agents are vulnerable to indirect prompt injection, in which malicious instructions embedded in untrusted data can trigger dangerous agent actions.<n>This position paper discusses our vision for system-level defenses against indirect prompt injection attacks.
  arXiv  Detail & Related papers  (2026-03-31T17:15:46Z)
• CourtGuard: A Model-Agnostic Framework for Zero-Shot Policy Adaptation in LLM Safety [8.24714635902347]
  We introduce CourtGuard, a retrieval-augmented multi-agent framework that reimagines safety evaluation as Evidentiary Debate.<n>By orchestrating an adversarial debate grounded in external policy documents, CourtGuard achieves state-of-the-art performance across 7 safety benchmarks.
  arXiv  Detail & Related papers  (2026-02-26T02:52:11Z)
• Spider-Sense: Intrinsic Risk Sensing for Efficient Agent Defense with Hierarchical Adaptive Screening [23.066685616914807]
  We argue that effective agent security should be intrinsic and selective rather than architecturally decoupled and mandatory.<n>We propose Spider-Sense framework, which allows agents to maintain latent vigilance and trigger defenses only upon risk perception.<n>Spider-Sense achieves competitive or superior defense performance, attaining the lowest Attack Success Rate (ASR) and False Positive Rate (FPR)
  arXiv  Detail & Related papers  (2026-02-05T07:11:05Z)
• BAPO: Boundary-Aware Policy Optimization for Reliable Agentic Search [72.87861928940929]
  Boundary-Aware Policy Optimization (BAPO) is a novel RL framework designed to cultivate reliable boundary awareness without compromising accuracy.<n>BAPO introduces two key components: (i) a group-based boundary-aware reward that encourages an IDK response only when the reasoning reaches its limit, and (ii) an adaptive reward modulator that strategically suspends this reward during early exploration, preventing the model from exploiting IDK as a shortcut.
  arXiv  Detail & Related papers  (2026-01-16T07:06:58Z)
• How Brittle is Agent Safety? Rethinking Agent Risk under Intent Concealment and Task Complexity [55.441602598245744]
  Current safety evaluations for LLM-driven agents primarily focus on atomic harms, failing to address sophisticated threats where malicious intent is concealed or diluted within complex tasks.<n>We address this gap with a two-dimensional analysis of agent safety brittleness under the pressures of intent concealment and task complexity.<n>Our findings reveal two critical phenomena: safety alignment degrades sharply and predictably as intent becomes obscured, and a "Complexity Paradox" emerges, where agents seem safer on harder tasks only due to capability limitations.
  arXiv  Detail & Related papers  (2025-11-11T17:27:27Z)
• Breaking Agent Backbones: Evaluating the Security of Backbone LLMs in AI Agents [36.2255033141489]
  AI agents powered by large language models (LLMs) are being deployed at scale, yet we lack a systematic understanding of how the choice of backbone LLM affects agent security.<n>We introduce threat snapshots: a framework that isolates specific states in an agent's execution flow where vulnerabilities manifest.<n>We apply this framework to construct the $operatornameb3$ benchmark, a security benchmark based on 194331 unique crowdsourced adversarial attacks.
  arXiv  Detail & Related papers  (2025-10-26T10:36:42Z)
• SecureWebArena: A Holistic Security Evaluation Benchmark for LVLM-based Web Agents [28.60912756104713]
  Large vision-language model (LVLM)-based web agents are emerging as powerful tools for automating complex online tasks.<n>Existing benchmarks provide only partial coverage, typically restricted to narrow scenarios.<n>We present tool, the first holistic benchmark for evaluating the security of LVLM-based web agents.
  arXiv  Detail & Related papers  (2025-10-11T07:18:12Z)
• Security Challenges in AI Agent Deployment: Insights from a Large Scale Public Competition [101.86739402748995]
  We run the largest public red-teaming competition to date, targeting 22 frontier AI agents across 44 realistic deployment scenarios.<n>We build the Agent Red Teaming benchmark and evaluate it across 19 state-of-the-art models.<n>Our findings highlight critical and persistent vulnerabilities in today's AI agents.
  arXiv  Detail & Related papers  (2025-07-28T05:13:04Z)
• OpenAgentSafety: A Comprehensive Framework for Evaluating Real-World AI Agent Safety [58.201189860217724]
  We introduce OpenAgentSafety, a comprehensive framework for evaluating agent behavior across eight critical risk categories.<n>Unlike prior work, our framework evaluates agents that interact with real tools, including web browsers, code execution environments, file systems, bash shells, and messaging platforms.<n>It combines rule-based analysis with LLM-as-judge assessments to detect both overt and subtle unsafe behaviors.
  arXiv  Detail & Related papers  (2025-07-08T16:18:54Z)
• AgentVigil: Generic Black-Box Red-teaming for Indirect Prompt Injection against LLM Agents [54.29555239363013]
  We propose a generic black-box fuzzing framework, AgentVigil, to automatically discover and exploit indirect prompt injection vulnerabilities.<n>We evaluate AgentVigil on two public benchmarks, AgentDojo and VWA-adv, where it achieves 71% and 70% success rates against agents based on o3-mini and GPT-4o.<n>We apply our attacks in real-world environments, successfully misleading agents to navigate to arbitrary URLs, including malicious sites.
  arXiv  Detail & Related papers  (2025-05-09T07:40:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.