Boiling the Frog: A Multi-Turn Benchmark for Agentic Safety

• URL: http://arxiv.org/abs/2605.22643v2
• Date: Fri, 22 May 2026 14:53:30 GMT
• Title: Boiling the Frog: A Multi-Turn Benchmark for Agentic Safety
• Abstract summary: We introduce Boiling the Frog, a benchmark that evaluates whether tool-using AI models are susceptible to incremental attacks.<n> Scenarios are organized through a three-level operational risk taxonomy.<n>Across a nine-model panel, aggregate strict attack success rate (ASR) is 44.4%.
• Score: 2.661610409070365
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Background. Traditional safety benchmarks for language models evaluate generated text: whether a model outputs toxic language, reproduces bias, or follows harmful instructions. When models are deployed as agents, the safety-relevant object shifts from what the system says to what it does within an environment, and evaluating model responses under prompting is no longer sufficient to address the safety challenges posed by artificial intelligence. Recent developments have seen the rise of benchmarks that evaluate large language models as agents. We contribute to this strand of research. Approach. We introduce Boiling the Frog, a benchmark that evaluates whether tool-using AI models deployed in corporate and office settings are susceptible to incremental attacks. Each scenario begins with benign workspace edits and later introduces a risk-bearing request. The benchmark focuses on stateful multi-turn evaluation: chains expose a persistent workspace, place the risk-bearing payload at controlled positions in the turn sequence, and score whether the resulting artifact state becomes unsafe. Scenarios are organized through a three-level operational risk taxonomy grounded in the Boiling the Frog risks, the AI Act Annex I and Annex III high-risk contexts, and EU AI Act's Code of Practice on General-Purpose AI (GPAI). Results. Across a nine-model panel, aggregate strict attack success rate (ASR) is 44.4%. Model-level ASR ranges from 20.5% for Claude Haiku 4.5 to 92.9% for Gemini 3.1 Flash Lite, with Seed 2.0 Lite also above 80%. Average chain category-level ASR reaches 93.3% for Code of Practice loss-of-control scenarios.

Related papers

• SingGuard-NSFA: Extensible Guardrails for Agentic AI via Generative Reasoning and Real-Time Classification [0.0]
  nsfaguard is a guardrail framework for securing agentic AI systems against operational threats.<n>We first introduce the NSFA taxonomy, which organizes 185 risk variants into a CIA-triad-grounded hierarchy.<n>We develop a dual-mode approach combining SFT-based generative reasoning for interpretable offline auditing with discriminative classification heads on the frozen backbone.
  arXiv  Detail & Related papers  (2026-07-13T07:28:45Z)
• Muse Spark Safety & Preparedness Report [106.21435337776768]
  Muse Spark is the latest large language model developed by Meta.<n>We first present evaluations for catastrophic risk domains under Meta's Advanced AI Scaling Framework.<n>We then discuss additional considerations, such as Muse Spark's broader content safety and behavioral profile.
  arXiv  Detail & Related papers  (2026-05-14T23:12:14Z)
• Emergent Strategic Reasoning Risks in AI: A Taxonomy-Driven Evaluation Framework [63.74295981594549]
  Large language models (LLMs) engage in behaviors that serve their own objectives.<n>These include, but are not limited to, deception (intentionally misleading users or evaluators), evaluation gaming (strategically manipulating performance during safety testing), and reward hacking.<n>We introduce ESRRSim, a taxonomy-driven agentic framework for automated behavioral risk evaluation.
  arXiv  Detail & Related papers  (2026-04-23T23:44:01Z)
• SafetyALFRED: Evaluating Safety-Conscious Planning of Multimodal Large Language Models [46.968844120076916]
  We introduce SafetyALFRED, built upon the embodied agent benchmark ALFRED, augmented with six categories of real-world kitchen hazards.<n>We evaluate eleven state-of-the-art models on not only hazard recognition, but also active risk mitigation through embodied planning.
  arXiv  Detail & Related papers  (2026-04-21T16:27:20Z)
• The Blind Spot of Agent Safety: How Benign User Instructions Expose Critical Vulnerabilities in Computer-Use Agents [29.98056047452529]
  Computer-use agents (CUAs) can now autonomously complete complex tasks in real environments, but when misled, they can also be automated.<n>We introduce OSBLIND, a benchmark that evaluates CUAs under unintended attack conditions.<n>Most CUAs exceed 90% attack success rate (ASR), and even safety-aligned Claude 4.5 Sonnet reaches 72.7% ASR.
  arXiv  Detail & Related papers  (2026-04-12T10:52:42Z)
• ATBench: A Diverse and Realistic Agent Trajectory Benchmark for Safety Evaluation and Diagnosis [96.92417622318267]
  ATBench is a trajectory-level benchmark for structured, diverse, and realistic evaluation of agent safety.<n>It organizes agentic risk along three dimensions: risk source, failure mode, and real-world harm.<n>It contains 1,000 trajectories (503 safe and 497 unsafe), averaging 9.01 turns and 3.95k tokens, with 1,954 invoked tools drawn from pools spanning 2,084 available tools.
  arXiv  Detail & Related papers  (2026-04-02T13:26:20Z)
• AutoControl Arena: Synthesizing Executable Test Environments for Frontier AI Risk Evaluation [22.833977971686966]
  We present AutoControl Arena, an automated framework for frontier AI risk evaluation.<n>We mitigate hallucination while maintaining flexibility.<n>This principle, instantiated through a three-agent framework, achieves over 98% end-to-end success and 60% human preference over existing simulators.
  arXiv  Detail & Related papers  (2026-03-08T02:49:45Z)
• ILION: Deterministic Pre-Execution Safety Gates for Agentic AI Systems [0.0]
  We present ILION, a deterministic execution gate for agentic AI systems.<n>ILION classifies proposed agent actions as BLOCK or ALLOW without statistical training or API dependencies.<n>ILION outperforms the best commercial baseline by 4.3 F1 points while operating 2,000 times faster with a false positive rate four times lower.
  arXiv  Detail & Related papers  (2026-02-22T12:25:42Z)
• ReasAlign: Reasoning Enhanced Safety Alignment against Prompt Injection Attack [52.17935054046577]
  We present ReasAlign, a model-level solution to improve safety alignment against indirect prompt injection attacks.<n>ReasAlign incorporates structured reasoning steps to analyze user queries, detect conflicting instructions, and preserve the continuity of the user's intended tasks.
  arXiv  Detail & Related papers  (2026-01-15T08:23:38Z)
• PropensityBench: Evaluating Latent Safety Risks in Large Language Models via an Agentic Approach [49.14349403242654]
  We present $textbfPropensityBench$, a novel benchmark framework that assesses the proclivity of models to engage in risky behaviors.<n>Our framework includes 5,874 scenarios with 6,648 tools spanning four high-risk domains: cybersecurity, self-proliferation, biosecurity, and chemical security.<n>Across open-source and proprietary frontier models, we uncover 9 alarming signs of propensity: models frequently choose high-risk tools when under pressure.
  arXiv  Detail & Related papers  (2025-11-24T18:46:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.