Taxonomy and Consistency Analysis of Safety Benchmarks for AI Agents

• URL: http://arxiv.org/abs/2605.16282v1
• Date: Sat, 11 Apr 2026 04:25:19 GMT
• Title: Taxonomy and Consistency Analysis of Safety Benchmarks for AI Agents
• Abstract summary: We present the first systematic analysis dedicated to agent safety benchmarks as evaluation instruments.<n>A coverage matrix reveals broad risk coverage but limited methodological convergence, while the taxonomy analysis shows a behavioral-benchmark core concentrated in sandboxed, constrained, and often safety-only evaluation.<n>Across the landscape, we find that benchmark choice can yield contradictory safety conclusions, coverage counts often overstate evaluation depth, environment fidelity shapes reported safety, the field disproportionately tests externally imposed rather than agent-internal risks, metric fragmentation limits comparison, and remains effectively unbenchmarked.
• Score: 6.787194586338237
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The rapid deployment of LLM-based autonomous agents has introduced safety risks that extend far beyond traditional LLM concerns, prompting a proliferation of safety benchmarks since late 2023. However, these benchmarks have developed independently, with inconsistent threat models, incompatible metrics, and overlapping yet incomplete risk coverage. We present the first systematic analysis dedicated to agent safety benchmarks as evaluation instruments. We catalog 40 behavioral agent-safety benchmarks (2023-2026), plus 5 adjacent evaluator, defense, and dataset artifacts, propose a six-axis taxonomy of benchmark evaluation methodology, and apply it across the corpus to characterize how methodological choices shape safety conclusions. A coverage matrix reveals broad risk coverage but limited methodological convergence, while the taxonomy analysis shows a behavioral-benchmark core concentrated in sandboxed, constrained, and often safety-only evaluation. Across the landscape, we find that benchmark choice can yield contradictory safety conclusions, coverage counts often overstate evaluation depth, environment fidelity systematically shapes reported safety, the field disproportionately tests externally imposed rather than agent-internal risks, metric fragmentation limits comparison, and robustness remains effectively unbenchmarked. We ground these claims with a cross-benchmark consistency check, with 95% confidence intervals and Kendall's W concordance analysis, finding no evidence of ranking concordance across evaluation dimensions (W = 0.10, p = 0.94). We release structured metadata, full taxonomy codings, risk annotations, and all experimental artifacts, and propose minimum reporting standards for future benchmarks.

Related papers

• SafeAtlas-VL: Beyond Binary Multimodal Safety with Large-Scale Data and Guard Models [73.04773649437375]
  Multimodal safety moderation requires distinguishing risks arising from visual content, user intent, and assistant behavior.<n>We introduce SafeAtlas-VL, a dataset of 1.5M training instances that places image-, request-, and response-level judgments on a five-level ordered scale.<n>We also construct SafeAtlas-Bench, a set of 5,000 instances for evaluating five-level predictions and continuous risk scores.
  arXiv  Detail & Related papers  (2026-08-29T07:08:01Z)
• Benchmarking the Benchmarks: Evaluating Automated Safety Benchmarks for Small Language Models [10.539135916505133]
  Small Language Models (SLMs) are increasingly deployed in resource-constrained, privacy-sensitive settings.<n>Existing AI safetyslash securityslash compliance benchmarks are designed for large language models that may not transfer reliably to SLMs.<n>We ask: Can these benchmarks effectively and reliably evaluate SLMs?
  arXiv  Detail & Related papers  (2026-08-17T22:47:14Z)
• Navigating the Sea of LLM Evaluation: Investigating Bias in Toxicity Benchmarks [0.36748639131154315]
  This work investigates the robustness of established benchmarking setups and examines how to measure intrinsic biases.<n>Our experiments uncover significant discrepancies in benchmark behaviors when evaluation setups are altered.<n>We observe model-specific instabilities, demonstrating a clear need for more robust and comprehensive safety evaluation frameworks.
  arXiv  Detail & Related papers  (2026-05-11T14:27:39Z)
• AISafetyBenchExplorer: A Metric-Aware Catalogue of AI Safety Benchmarks Reveals Fragmented Measurement and Weak Benchmark Governance [0.0]
  We present AISafetyBenchExplorer, a structured catalogue of 195 AI safety benchmarks released between 2018 and 2026.<n>We identify a central structural problem: benchmark proliferation has outpaced measurement standardization.<n>At the metric level, the catalogue shows that familiar labels such as accuracy, F1 score, safety score, and aggregate benchmark scores often conceal materially different judges, aggregation rules, and threat models.
  arXiv  Detail & Related papers  (2026-04-14T15:26:03Z)
• ATBench: A Diverse and Realistic Agent Trajectory Benchmark for Safety Evaluation and Diagnosis [96.92417622318267]
  ATBench is a trajectory-level benchmark for structured, diverse, and realistic evaluation of agent safety.<n>It organizes agentic risk along three dimensions: risk source, failure mode, and real-world harm.<n>It contains 1,000 trajectories (503 safe and 497 unsafe), averaging 9.01 turns and 3.95k tokens, with 1,954 invoked tools drawn from pools spanning 2,084 available tools.
  arXiv  Detail & Related papers  (2026-04-02T13:26:20Z)
• TeleAI-Safety: A comprehensive LLM jailbreaking benchmark towards attacks, defenses, and evaluations [46.98103424349245]
  Existing safety evaluation benchmarks and frameworks are often limited by an imbalanced integration of core components.<n>We present TeleAI-Safety, a modular and reproducible framework coupled with a systematic benchmark for rigorous safety evaluation.<n>With a curated attack corpus of 342 samples spanning 12 distinct risk categories, the TeleAI-Safety benchmark conducts extensive evaluations across 14 target models.
  arXiv  Detail & Related papers  (2025-12-05T07:23:30Z)
• SafeRBench: A Comprehensive Benchmark for Safety Assessment in Large Reasoning Models [60.8821834954637]
  We present SafeRBench, the first benchmark that assesses LRM safety end-to-end.<n>We pioneer the incorporation of risk categories and levels into input design.<n>We introduce a micro-thought chunking mechanism to segment long reasoning traces into semantically coherent units.
  arXiv  Detail & Related papers  (2025-11-19T06:46:33Z)
• Advancing Embodied Agent Security: From Safety Benchmarks to Input Moderation [52.83870601473094]
  Embodied agents exhibit immense potential across a multitude of domains.<n>Existing research predominantly concentrates on the security of general large language models.<n>This paper introduces a novel input moderation framework, meticulously designed to safeguard embodied agents.
  arXiv  Detail & Related papers  (2025-04-22T08:34:35Z)
• SeCodePLT: A Unified Platform for Evaluating the Security of Code GenAI [58.29510889419971]
  Existing benchmarks for evaluating the security risks and capabilities of code-generating large language models (LLMs) face several key limitations.<n>We introduce a general and scalable benchmark construction framework that begins with manually validated, high-quality seed examples and expands them via targeted mutations.<n>Applying this framework to Python, C/C++, and Java, we build SeCodePLT, a dataset of more than 5.9k samples spanning 44 CWE-based risk categories and three security capabilities.
  arXiv  Detail & Related papers  (2024-10-14T21:17:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.