Who Tests the Testers? Systematic Enumeration and Coverage Audit of LLM Agent Tool Call Safety

• URL: http://arxiv.org/abs/2603.18245v1
• Date: Wed, 18 Mar 2026 20:06:47 GMT
• Title: Who Tests the Testers? Systematic Enumeration and Coverage Audit of LLM Agent Tool Call Safety
• Abstract summary: Large Language Model (LLM) agents increasingly act through external tools, making their safety contingent on tool-call rather than text generation alone.<n>While recent benchmarks evaluate agents across diverse environments and risk categories, a fundamental question remains unanswered: how complete are existing test suites, and what unsafe interaction patterns persist even after an agent passes the benchmark?<n>We propose SafeAudit, a meta-audit framework that addresses this gap through two contributions. First, an enumerator that systematically generates test cases by enumerating valid tool-call and diverse user scenarios. Second, we introduce rule-resistance, a non-semantic, quantitative metric
• Score: 32.40159096400358
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large Language Model (LLM) agents increasingly act through external tools, making their safety contingent on tool-call workflows rather than text generation alone. While recent benchmarks evaluate agents across diverse environments and risk categories, a fundamental question remains unanswered: how complete are existing test suites, and what unsafe interaction patterns persist even after an agent passes the benchmark? We propose SafeAudit, a meta-audit framework that addresses this gap through two contributions. First, an LLM-based enumerator that systematically generates test cases by enumerating valid tool-call workflows and diverse user scenarios. Second, we introduce rule-resistance, a non-semantic, quantitative metric that distills compact safety rules from existing benchmarks and identifies unsafe interaction patterns that remain uncovered under those rules. Across 3 benchmarks and 12 environments, SafeAudit uncovers more than 20% residual unsafe behaviors that existing benchmarks fail to expose, with coverage growing monotonically as the testing budget increases. Our results highlight significant completeness gaps in current safety evaluation and motivate meta-auditing as a necessary complement to benchmark-based agent safety testing.

Related papers

• Safety Testing LLM Agents at Scale: From Risk Discovery to Evidence-Grounded Verification [32.337075755765056]
  Vera is an endto-end automated safety testing framework that instantiates software engineering testing principles for non-deterministic agents.<n>We release Vera, comprising 1600 executable safety cases spanning 124 risk categories across execution settings.<n>These results indicate that modular, executable testing infrastructure is essential for rigorous and maintainable safety evaluation of rapidly evolving agentic systems.
  arXiv  Detail & Related papers  (2026-07-02T07:08:26Z)
• SeClaw: Spec-Driven Security Task Synthesis for Evaluating Autonomous Agents [87.26967184869198]
  SeClaw is a framework that combines specification-driven security task synthesis with execution-based security evaluation for Autonomous agents.<n> benchmark covers risks arising from resources, user tasks, environments, and intrinsic agent behaviors.
  arXiv  Detail & Related papers  (2026-06-01T14:23:42Z)
• A Unified Framework for the Evaluation of LLM Agentic Capabilities [36.43241368835721]
  We present a unified framework for the fair evaluation of LLM agentic capabilities.<n> Driven by a unified configuration system, the framework integrates diverse benchmarks into a standardized instruction--tool--environment format.<n>We conduct a large-scale empirical analysis over 400K rollouts and 5B tokens on 15 models.
  arXiv  Detail & Related papers  (2026-05-27T03:20:45Z)
• Navigating the Sea of LLM Evaluation: Investigating Bias in Toxicity Benchmarks [0.36748639131154315]
  This work investigates the robustness of established benchmarking setups and examines how to measure intrinsic biases.<n>Our experiments uncover significant discrepancies in benchmark behaviors when evaluation setups are altered.<n>We observe model-specific instabilities, demonstrating a clear need for more robust and comprehensive safety evaluation frameworks.
  arXiv  Detail & Related papers  (2026-05-11T14:27:39Z)
• Robust Multimodal Safety via Conditional Decoding [52.92816441364308]
  Multimodal large-language models (MLLMs) often experience degraded safety alignment when harmful queries exploit cross-modal interactions.<n>We propose a simple conditional decoding strategy, CASA, that utilizes internal representations of MLLMs to predict a binary safety token before response generation.
  arXiv  Detail & Related papers  (2026-03-31T23:19:50Z)
• RewardHackingAgents: Benchmarking Evaluation Integrity for LLM ML-Engineering Agents [0.9821874476902969]
  LLM agents increasingly perform end-to-end ML engineering tasks where success is judged by a single scalar test metric.<n>This creates a structural vulnerability: an agent can increase the reported score by compromising the evaluation pipeline rather than improving the model.<n>We introduce RewardHackingAgents, a workspace-based benchmark that makes two compromise vectors explicit and measurable.
  arXiv  Detail & Related papers  (2026-03-11T22:06:44Z)
• Unsafer in Many Turns: Benchmarking and Defending Multi-Turn Safety Risks in Tool-Using Agents [68.20752678837377]
  We propose a principled taxonomy that transforms single-turn harmful tasks into multi-turn attack sequences.<n>Using this taxonomy, we construct MT-AgentRisk, the first benchmark to evaluate multi-turn tool-using agent safety.<n>We propose ToolShield, a training-free, tool-agnostic, self-exploration defense.
  arXiv  Detail & Related papers  (2026-02-13T18:38:18Z)
• RealSec-bench: A Benchmark for Evaluating Secure Code Generation in Real-World Repositories [58.32028251925354]
  Large Language Models (LLMs) have demonstrated remarkable capabilities in code generation, but their proficiency in producing secure code remains a critical, under-explored area.<n>We introduce RealSec-bench, a new benchmark for secure code generation meticulously constructed from real-world, high-risk Java repositories.
  arXiv  Detail & Related papers  (2026-01-30T08:29:01Z)
• DUALGUAGE: Automated Joint Security-Functionality Benchmarking for Secure Code Generation [18.208257676875586]
  Large language models (LLMs) and autonomous coding agents are increasingly used to generate software across a wide range of unmet domains.<n>Existing benchmarks and evaluations for secure code generation fall short-many measure only vulnerability reduction, disregard correctness preservation, or evaluate security and functionality on separate datasets.<n>We present GAUGE, the first fully automated benchmarking framework designed to rigorously evaluate the security and correctness of LLM-generated code in unison.
  arXiv  Detail & Related papers  (2025-11-24T22:26:14Z)
• STAF: Leveraging LLMs for Automated Attack Tree-Based Security Test Generation [3.283937504286784]
  This paper introduces STAF (Security Test Automation Framework), a novel approach to automating security test case generation.<n>We show the elements and processes needed to produce sensible and executable automotive security test suites, along with the integration with an automated testing framework.
  arXiv  Detail & Related papers  (2025-09-24T14:46:42Z)
• AgentAuditor: Human-Level Safety and Security Evaluation for LLM Agents [48.925168866726814]
  AgentAuditor is a universal, training-free, memory-augmented reasoning framework.<n>ASSEBench is the first benchmark designed to check how well LLM-based evaluators can spot both safety risks and security threats.
  arXiv  Detail & Related papers  (2025-05-31T17:10:23Z)
• ASTRAL: Automated Safety Testing of Large Language Models [6.1050306667733185]
  Large Language Models (LLMs) have recently gained attention due to their ability to understand and generate sophisticated human-like content.<n>We present ASTRAL, a tool that automates the generation and execution of test cases (i.e., prompts) for testing the safety of LLMs.
  arXiv  Detail & Related papers  (2025-01-28T18:25:11Z)
• Agent-SafetyBench: Evaluating the Safety of LLM Agents [72.92604341646691]
  We introduce Agent-SafetyBench, a benchmark designed to evaluate the safety of large language models (LLMs)<n>Agent-SafetyBench encompasses 349 interaction environments and 2,000 test cases, evaluating 8 categories of safety risks and covering 10 common failure modes frequently encountered in unsafe interactions.<n>Our evaluation of 16 popular LLM agents reveals a concerning result: none of the agents achieves a safety score above 60%.
  arXiv  Detail & Related papers  (2024-12-19T02:35:15Z)
• SafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM Agents [58.65256663334316]
  We present SafeAgentBench -- the first benchmark for safety-aware task planning of embodied LLM agents in interactive simulation environments.<n>SafeAgentBench includes: (1) an executable, diverse, and high-quality dataset of 750 tasks, rigorously curated to cover 10 potential hazards and 3 task types; (2) SafeAgentEnv, a universal embodied environment with a low-level controller, supporting multi-agent execution with 17 high-level actions for 9 state-of-the-art baselines; and (3) reliable evaluation methods from both execution and semantic perspectives.
  arXiv  Detail & Related papers  (2024-12-17T18:55:58Z)
• BELLS: A Framework Towards Future Proof Benchmarks for the Evaluation of LLM Safeguards [43.86118338226387]
  We introduce the Benchmarks for the Evaluation of LLM Safeguards (BELLS) BELLS is a structured collection of tests, organized into three categories: established failure tests, emerging failure tests and next-gen architecture tests. We implement and share the first next-gen architecture test, using the MACHIAVELLI environment, along with an interactive visualization of the dataset.
  arXiv  Detail & Related papers  (2024-06-03T14:32:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.