AgentTrust: Runtime Safety Evaluation and Interception for AI Agent Tool Use

• URL: http://arxiv.org/abs/2605.04785v1
• Date: Wed, 06 May 2026 11:38:16 GMT
• Title: AgentTrust: Runtime Safety Evaluation and Interception for AI Agent Tool Use
• Abstract summary: AgentTrust intercepts agent tool calls before execution and returns a structured verdict: allow, warn, block, or review.<n>We release a 300-scenario benchmark across six risk categories and an additional 630 independently constructed real-world adversarial scenarios.<n>On the 630-second benchmark, evaluated under a patched ruleset, AgentTrust achieves 96.7% verdict accuracy, including about 93% on shell-obfuscated payloads.
• Score: 2.9991161518367875
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Modern AI agents execute real-world side effects through tool calls such as file operations, shell commands, HTTP requests, and database queries. A single unsafe action, including accidental deletion, credential exposure, or data exfiltration, can cause irreversible harm. Existing defenses are incomplete: post-hoc benchmarks measure behavior after execution, static guardrails miss obfuscation and multi-step context, and infrastructure sandboxes constrain where code runs without understanding what an action means. We present AgentTrust, a runtime safety layer that intercepts agent tool calls before execution and returns a structured verdict: allow, warn, block, or review. AgentTrust combines a shell deobfuscation normalizer, SafeFix suggestions for safer alternatives, RiskChain detection for multi-step attack chains, and a cache-aware LLM-as-Judge for ambiguous inputs. We release a 300-scenario benchmark across six risk categories and an additional 630 independently constructed real-world adversarial scenarios. On the internal benchmark, the production-only ruleset achieves 95.0% verdict accuracy and 73.7% risk-level accuracy at low-millisecond end-to-end latency. On the 630-scenario benchmark, evaluated under a patched ruleset and not claimed as zero-shot, AgentTrust achieves 96.7% verdict accuracy, including about 93% on shell-obfuscated payloads. AgentTrust is released under the AGPL-3.0 license and provides a Model Context Protocol server for MCP-compatible agents.

Related papers

• AgentAudit: An Open, Extensible Framework for Full-Lifecycle Trust Evaluation of AI Agents [2.8000808592516293]
  AgentAudit evaluates the entire execution trace across ten capability, grounding, security and behavioural dimensions.<n>It reads only the recorded execution trace and does not interfere with how the agent runs.<n>We evaluate five language models (OpenAI GPT-5, Claude Sonnet 5, Sarvam 105B, Llama 3.3 70B and Gemini 2.5 Flash) across nine capability and adversarial tasks.
  arXiv  Detail & Related papers  (2026-09-09T08:29:16Z)
• ROPE: Routed Origin Policy Enforcement against Indirect Prompt Injection [61.4148196085256]
  We present ROPE (Routed Origin Policy Enforcement), which is anchored in a structural notion of trust.<n>Our approach admits two provable guarantees: 1) at every step of a trajectory, no value whose only origin is attacker-writable content reaches an origin-guarded parameter, and 2) no rewording of an injection changes an admission decision.
  arXiv  Detail & Related papers  (2026-08-27T01:22:14Z)
• LMSM: LLM Security Framework Inspired by Linux Security Modules [56.93644694627783]
  Large language models (LLMs) are increasingly deployed with layered defenses, yet malicious prompts can still bypass them.<n>We present Language Model Security Modules (LMSM), a security framework that adapts the separation behind Linux Security Modules (LSM) to LLM serving.
  arXiv  Detail & Related papers  (2026-08-26T12:13:05Z)
• StepGuard: Learning Step-Level Guardrails with Scalable Supervision and Safety-Utility Balancing [80.39644823192113]
  We propose StepGuard, a step-level guard model that can audit completed agent trajectories and check tool actions before they are executed.<n>StepGuard achieves the highest average accuracy among open-weight guard models, with performance comparable to GPT-5.4.<n>When used to guard agents on AgentDojo and AgentDyn, StepGuard reduces mean attack success rate by 77.3%.
  arXiv  Detail & Related papers  (2026-08-25T16:17:27Z)
• AgentFlow: A Flow-Centric Policy Language and Framework for Securing LLM Agent Systems [7.734540679538682]
  LLM agents increasingly read untrusted content, invoke external tools, access private data, and delegate work to other agents.<n>We present AgentFlow, a flow-centric policy language and runtime enforcement model for specifying where data may travel in agent systems.
  arXiv  Detail & Related papers  (2026-08-24T06:56:00Z)
• ClawProBench: Trace-Aware Evaluation of AI Agents with Runtime Coverage and Frozen Workplace-Style Holdouts [0.0]
  Agent benchmarks often evaluate only final answers even when agents run on stateful runtimes.<n>We present ClawProBench, a trace-aware benchmark for runtime-native agent evaluation instantiated on OpenClaw.
  arXiv  Detail & Related papers  (2026-08-23T17:09:02Z)
• ClawSentry: A Progressive Multi-Tier Security Monitor for Safeguarding Autonomous LLM Agents [24.540638169054812]
  A single agent hijacked by a malicious third-party skill can cause data exfiltration, privilege escalation, or cascading compromise.<n>We present ClawSentry, an open-source, framework-agnostic security supervision gateway for agent runtimes.
  arXiv  Detail & Related papers  (2026-08-21T13:47:51Z)
• Understanding and Evaluating Claw-like Agent Security Through a Computer-Systems Lens [65.53494487819053]
  Claw-like AI agents (e.g., OpenClaw) are always-on processes with persistent access to credentials, files, tools, and external services.<n>We adopt a computer-system analogy: treating a Claw-like agent as an agentic computer system whose gateway runtime plays an OS-like mediation role.<n>We develop SafeClawArena, a benchmark of 406 adversarial tasks across four attack surfaces.
  arXiv  Detail & Related papers  (2026-06-29T18:00:45Z)
• ClawSafety: "Safe" LLMs, Unsafe Agents [25.729388843970014]
  Personal AI agents like OpenClaw run with elevated privileges on users' local machines.<n>ClawSAFETY is a benchmark of 120 adversarial test scenarios organized along three dimensions (harm domain, attack vector, and harmful action type)<n>Each test case embeds adversarial content in one of three channels the agent encounters during normal work: workspace skill files, emails from trusted senders, and web pages.
  arXiv  Detail & Related papers  (2026-04-01T22:24:24Z)
• AgentSentry: Mitigating Indirect Prompt Injection in LLM Agents via Temporal Causal Diagnostics and Context Purification [25.817251923574286]
  We propose a novel inference-time detection and mitigation framework for large language model (LLM) agents.<n>AgentSentry is the first inference-time defense to model multi-turn IPI as a temporal causal takeover.<n>We evaluate AgentSentry on the textscAgentDojo benchmark across four task suites, three IPI attack families, and multiple black-box LLMs.
  arXiv  Detail & Related papers  (2026-02-26T07:59:10Z)
• ReasAlign: Reasoning Enhanced Safety Alignment against Prompt Injection Attack [52.17935054046577]
  We present ReasAlign, a model-level solution to improve safety alignment against indirect prompt injection attacks.<n>ReasAlign incorporates structured reasoning steps to analyze user queries, detect conflicting instructions, and preserve the continuity of the user's intended tasks.
  arXiv  Detail & Related papers  (2026-01-15T08:23:38Z)
• BackdoorAgent: A Unified Framework for Backdoor Attacks on LLM-based Agents [58.83028403414688]
  Large language model (LLM) agents execute tasks through multi-step workflow that combine planning, memory, and tool use.<n>Backdoor triggers injected into specific stages of an agent workflow can persist through multiple intermediate states and adversely influence downstream outputs.<n>We propose textbfBackdoorAgent, a modular and stage-aware framework that provides a unified agent-centric view of backdoor threats in LLM agents.
  arXiv  Detail & Related papers  (2026-01-08T03:49:39Z)
• Breaking the Code: Security Assessment of AI Code Agents Through Systematic Jailbreaking Attacks [11.371490212283383]
  Code-capable large language model (LLM) agents are embedded into software engineering where they can read, write, and execute code.<n>We present JAWS-BENCH, a benchmark spanning three escalating workspaces that mirror attacker capability.<n>We find that under prompt-only conditions in JAWS-0, code agents accept 61% of attacks on average; 58% are harmful, 52% parse, and 27% run end-to-end.
  arXiv  Detail & Related papers  (2025-10-01T18:38:20Z)
• VulAgent: Hypothesis-Validation based Multi-Agent Vulnerability Detection [55.957275374847484]
  VulAgent is a multi-agent vulnerability detection framework based on hypothesis validation.<n>It implements a semantics-sensitive, multi-view detection pipeline, each aligned to a specific analysis perspective.<n>On average, VulAgent improves overall accuracy by 6.6%, increases the correct identification rate of vulnerable--fixed code pairs by up to 450%, and reduces the false positive rate by about 36%.
  arXiv  Detail & Related papers  (2025-09-15T02:25:38Z)
• OpenAgentSafety: A Comprehensive Framework for Evaluating Real-World AI Agent Safety [58.201189860217724]
  We introduce OpenAgentSafety, a comprehensive framework for evaluating agent behavior across eight critical risk categories.<n>Unlike prior work, our framework evaluates agents that interact with real tools, including web browsers, code execution environments, file systems, bash shells, and messaging platforms.<n>It combines rule-based analysis with LLM-as-judge assessments to detect both overt and subtle unsafe behaviors.
  arXiv  Detail & Related papers  (2025-07-08T16:18:54Z)
• AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents [84.96249955105777]
  LLM agents may pose a greater risk if misused, but their robustness remains underexplored.<n>We propose a new benchmark called AgentHarm to facilitate research on LLM agent misuse.<n>We find leading LLMs are surprisingly compliant with malicious agent requests without jailbreaking.
  arXiv  Detail & Related papers  (2024-10-11T17:39:22Z)
• Dissecting Adversarial Robustness of Multimodal LM Agents [70.2077308846307]
  We manually create 200 targeted adversarial tasks and evaluation scripts in a realistic threat model on top of VisualWebArena.<n>We find that we can successfully break latest agents that use black-box frontier LMs, including those that perform reflection and tree search.<n>We also use ARE to rigorously evaluate how the robustness changes as new components are added.
  arXiv  Detail & Related papers  (2024-06-18T17:32:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.