SLEIGHT-Bench: A Benchmark of Evasion Attacks Against Agent Monitors

• URL: http://arxiv.org/abs/2605.16626v2
• Date: Tue, 19 May 2026 00:46:42 GMT
• Title: SLEIGHT-Bench: A Benchmark of Evasion Attacks Against Agent Monitors
• Abstract summary: We present SLEIGHT-Bench (Subtle Low-itEration Insight-Guided Harmful Transcripts), a benchmark of synthetic transcripts containing 40 attacks across 11 categories.<n>We find that 20 of 40 attacks are never caught by an Opus 4.6 monitor with extended thinking across 10 trials at a 1% false-positive rate, while the overall catch rate is 32%.
• Score: 3.989565392866092
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Since autonomous coding agents generate complex behaviors at high-volume, we may want to use other LLMs to monitor actions to reduce the risk from dangerous misaligned behavior. To better understand the limitations of such monitors against the diverse attack strategies that a coding agent could use, we present SLEIGHT-Bench (Subtle Low-itEration Insight-Guided Harmful Transcripts), a benchmark of synthetic transcripts containing 40 attacks across 11 categories, each showing a coding agent covertly pursuing a harmful objective (e.g. weight exfiltration, credential theft, unmonitored rogue deployment). We find that 20 of 40 attacks are never caught by an Opus 4.6 monitor with extended thinking across 10 trials at a 1% false-positive rate, while the overall catch rate is 32%. We identify several evasion strategies such as exploiting model priors, user instruction ambiguity, and state manipulation, which facilitate creating attacks that evade frontier monitors. We also elicit stronger monitor performance using coding agents as monitors versus regular prompted monitors, and for some evasion strategies show improved catch rates with targeted monitor prompts. Our dataset and evaluation framework are available at https://github.com/safety-research/sleight-bench and https://huggingface.co/datasets/sleightbench/SLEIGHT-Bench.

Related papers

• Corrupt Plans, Clean Traces: Evading Chain-of-Thought Monitoring with Plan Injection [21.905085829126666]
  Chain-of-thought (CoT) monitoring is a safety strategy where the reasoning of a large language model "actor" is inspected by a "monitor"<n>We find that planting harmful but benign-sounding reasoning in the actor's context can steer it to perform adversarial actions while evading monitors.
  arXiv  Detail & Related papers  (2026-09-14T18:33:27Z)
• SIR: Self-improving Red-teaming for Compute Use Agents [71.71987044117371]
  Computer use agents (CUAs) are vision-language models that perceive a screen and act on a real operating system through mouse, keyboard, and terminal.<n>Because they can be exposed to untrusted content while operating, they are vulnerable to indirect prompt injection (IPI)<n>We present SIR, a black box IPI attack that composes stealthy injections from a small library of reusable principles stated in plain language.
  arXiv  Detail & Related papers  (2026-08-31T03:39:43Z)
• Distributed Attacks in Persistent-State AI Control [0.8411355178908656]
  Iterative VibeCoding is the study of safely deploying capable but potentially untrusted AI.<n>Evasion remains high across state-of-the-art monitor models.<n>We introduce a stateful link-tracker monitor that tracks suspicious buildup across PRs.
  arXiv  Detail & Related papers  (2026-07-02T17:59:56Z)
• Stateful Online Monitoring Catches Distributed Agent Attacks [63.00394432922707]
  We develop an online stateful monitor that uses real-time clustering to collect weak suspiciousness signals across many agent transcripts.<n>Our results point toward a new class of safety monitors which reason over groups of users rather than isolated transcripts.
  arXiv  Detail & Related papers  (2026-05-29T17:57:00Z)
• MonitoringBench: Semi-Automated Red-Teaming for Agent Monitoring [0.0]
  We introduce a red-teaming methodology that exposes harder-to-catch attacks for monitoring agents.<n>We find that frontier monitors often detect suspicious actions but fall for persuasion or fail to calibrate suspiciousness scores appropriately.<n>MonitoringBench is a benchmark of 2,644 attack trajectories for evaluating monitor capabilities and failure modes.
  arXiv  Detail & Related papers  (2026-05-10T18:04:31Z)
• TraceGuard: Structured Multi-Dimensional Monitoring as a Collusion-Resistant Control Protocol [1.0195618602298684]
  TraceGuard is a structured multi-dimensional monitoring protocol that evaluates agent actions across five dimensions.<n>It is implemented as a new monitor type for the open-source ControlArena framework.
  arXiv  Detail & Related papers  (2026-04-05T05:05:59Z)
• Bypassing AI Control Protocols via Agent-as-a-Proxy Attacks [12.356708678431183]
  Current defenses rely on monitoring protocols that jointly evaluate an agent's Chain-of-Thought (CoT) and tool-use actions to ensure alignment with user intent.<n>We demonstrate that these monitoring-based defenses can be bypassed via a novel Agent-as-a- Proxy attack.<n>Our findings suggest current monitoring-based agentic defenses are fundamentally fragile regardless of model scale.
  arXiv  Detail & Related papers  (2026-02-04T21:38:38Z)
• Beyond Linear Probes: Dynamic Safety Monitoring for Language Models [67.15793594651609]
  Traditional safety monitors require the same amount of compute for every query.<n>We introduce Truncated Polynomials (TPCs), a natural extension of linear probes for dynamic activation monitoring.<n>Our key insight is that TPCs can be trained and evaluated progressively, term-by-term.
  arXiv  Detail & Related papers  (2025-09-30T13:32:59Z)
• BlindGuard: Safeguarding LLM-based Multi-Agent Systems under Unknown Attacks [58.959622170433725]
  BlindGuard is an unsupervised defense method that learns without requiring any attack-specific labels or prior knowledge of malicious behaviors.<n>We show that BlindGuard effectively detects diverse attack types (i.e., prompt injection, memory poisoning, and tool attack) across multi-agent systems.
  arXiv  Detail & Related papers  (2025-08-11T16:04:47Z)
• Monitoring Decomposition Attacks in LLMs with Lightweight Sequential Monitors [27.976136688947093]
  Current LLM safety defenses fail under decomposition attacks, where a malicious goal is decomposed into benign subtasks that circumvent refusals.<n>We propose adding an external monitor that observes the conversation at a higher granularity.<n>We show that a carefully prompt engineered lightweight monitor achieves a 93% defense success rate, beating reasoning models like o3 mini as a monitor.
  arXiv  Detail & Related papers  (2025-06-12T17:50:58Z)
• Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation [56.102976602468615]
  We show that we can monitor a frontier reasoning model, such as OpenAI o3-mini, for reward hacking in agentic coding environments.<n>We find that with too much optimization, agents learn obfuscated reward hacking, hiding their intent within the chain-of-thought.
  arXiv  Detail & Related papers  (2025-03-14T23:50:34Z)
• Dissecting Adversarial Robustness of Multimodal LM Agents [70.2077308846307]
  We manually create 200 targeted adversarial tasks and evaluation scripts in a realistic threat model on top of VisualWebArena.<n>We find that we can successfully break latest agents that use black-box frontier LMs, including those that perform reflection and tree search.<n>We also use ARE to rigorously evaluate how the robustness changes as new components are added.
  arXiv  Detail & Related papers  (2024-06-18T17:32:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.