Benchmarks for Trajectory Safety Evaluation and Diagnosis in OpenClaw and Codex: ATBench-Claw and ATBench-CodeX

• URL: http://arxiv.org/abs/2604.14858v1
• Date: Thu, 16 Apr 2026 10:45:13 GMT
• Title: Benchmarks for Trajectory Safety Evaluation and Diagnosis in OpenClaw and Codex: ATBench-Claw and ATBench-CodeX
• Abstract summary: ATBench is a diverse and realistic agent trajectory benchmark for safety evaluation and diagnosis.<n>This report presents ATBench-Claw and ATBench-CodeX, two domain-customized extensions that carry ATBench into the OpenClaw and OpenAI Codex / Codex-runtime settings.
• Score: 54.138213405601896
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: As agent systems move into increasingly diverse execution settings, trajectory-level safety evaluation and diagnosis require benchmarks that evolve with them. ATBench is a diverse and realistic agent trajectory benchmark for safety evaluation and diagnosis. This report presents ATBench-Claw and ATBench-CodeX, two domain-customized extensions that carry ATBench into the OpenClaw and OpenAI Codex / Codex-runtime settings. The key adaptation mechanism is to analyze each new setting, customize the three-dimensional Safety Taxonomy over risk source, failure mode, and real-world harm, and then use that customized taxonomy to define the benchmark specification consumed by the shared ATBench construction pipeline. This extensibility matters because agent frameworks remain relatively stable at the architectural level even as their concrete execution settings, tool ecosystems, and product capabilities evolve quickly. Concretely, ATBench-Claw targets OpenClaw-sensitive execution chains over tools, skills, sessions, and external actions, while ATBench-CodeX targets trajectories in the OpenAI Codex / Codex-runtime setting over repositories, shells, patches, dependencies, approvals, and runtime policy boundaries. Our emphasis therefore falls on taxonomy customization, domain-specific risk coverage, and benchmark design under a shared ATBench generation framework.

Related papers

• HazardAuditor: From Executable Threats to Safer Computer-Use Agents [50.06351661912873]
  Existing guard models target static prompts and responses.<n>Existing safety platforms produce evaluation verdicts rather than the normalized supervision a guard model needs to learn.<n>We introduce HazardAuditor, an execution-grounded framework that closes both gaps.
  arXiv  Detail & Related papers  (2026-09-14T07:09:33Z)
• ICAE-Bench: Evaluating Coding Agents as Interactive Project Builders [26.859032352714276]
  We introduce ICAE-Bench, a benchmark for evaluating coding agents under interactive project-building settings.<n>The basic idea is to start from a fuzzy product requirement, simulating the dynamic paradigm with an automated User Agent.<n>ICAE-Bench uses standardized black-box tests together with multi-dimensional diagnostics, including functional correctness, semantic and API similarity, structural fidelity, design quality, and interaction quality.
  arXiv  Detail & Related papers  (2026-07-23T11:31:38Z)
• RealClawBench: Live OpenClaw Benchmarks from Real Developer-Agent Sessions [9.31655300350697]
  We introduce RealClawBench, a benchmark framework built from real OpenClaw sessions to capture the distribution, diversity, and real-world difficulty of deployed agent use.<n>RealClawBench addresses these challenges with two core mechanisms: reconstructed execution environments and deterministic verifiable scorers.<n>The resulting release contains 281 executable tasks sampled from a much larger real-session pool.
  arXiv  Detail & Related papers  (2026-06-02T16:51:24Z)
• ClawGym: A Scalable Framework for Building Effective Claw Agents [47.47444724291439]
  We present ClawGym, a scalable framework that supports the full lifecycle of Claw-style personal agent development.<n>ClawGym-SynData is a dataset of 13.5K filtered tasks synthesized from persona-driven intents and skill-grounded operations.<n>We then train a family of capable Claw-style models, termed ClawGym-Agents, through supervised fine-tuning on black-box rollout trajectories.<n>We further construct ClawGym-Bench, a benchmark of 200 instances through automated filtering and human-LLM review.
  arXiv  Detail & Related papers  (2026-04-29T17:12:22Z)
• ATBench: A Diverse and Realistic Agent Trajectory Benchmark for Safety Evaluation and Diagnosis [96.92417622318267]
  ATBench is a trajectory-level benchmark for structured, diverse, and realistic evaluation of agent safety.<n>It organizes agentic risk along three dimensions: risk source, failure mode, and real-world harm.<n>It contains 1,000 trajectories (503 safe and 497 unsafe), averaging 9.01 turns and 3.95k tokens, with 1,954 invoked tools drawn from pools spanning 2,084 available tools.
  arXiv  Detail & Related papers  (2026-04-02T13:26:20Z)
• AEC-Bench: A Multimodal Benchmark for Agentic Systems in Architecture, Engineering, and Construction [1.1841228164483055]
  The AEC-Bench is a benchmark for evaluating agentic systems on real-world tasks.<n>This report describes the benchmark motivation, dataset taxonomy, evaluation protocol, and baseline results.
  arXiv  Detail & Related papers  (2026-03-31T03:10:28Z)
• Clawed and Dangerous: Can We Trust Open Agentic Systems? [12.04979073308511]
  Open agentic systems combine planning with external capabilities, persistent memory, and privileged execution.<n>This paper introduces a six-dimensional analytical taxonomy and synthesizes 50 papers spanning attacks, benchmarks, defenses, audits, and adjacent engineering foundations.<n>Our review shows that the literature is relatively mature in attack characterization and benchmark construction, but remains weak in deployment controls, operational governance, persistent-memory integrity, and capability revocation.
  arXiv  Detail & Related papers  (2026-03-27T09:45:12Z)
• Symfrog-512: High-Capacity Sponge-Based AEAD Cipher (1024-bit State) [0.0]
  This submission includes a complete reference implementation together with deterministic test vectors and a reproducible benchmark suite.<n>The AEAD construction is fully specified, including domain separation, rate and capacity choices, tag generation, and the exact file format used by the reference CLI.<n>Reported performance numbers are produced by the built in benchmark tool under documented hardware and compiler settings.
  arXiv  Detail & Related papers  (2026-02-19T23:39:54Z)
• ABC-Bench: Benchmarking Agentic Backend Coding in Real-World Development [72.4729759618632]
  We introduce ABC-Bench, a benchmark to evaluate agentic backend coding within a realistic, executable workflow.<n>We curated 224 practical tasks spanning 8 languages and 19 frameworks from open-source repositories.<n>Our evaluation reveals that even state-of-the-art models struggle to deliver reliable performance on these holistic tasks.
  arXiv  Detail & Related papers  (2026-01-16T08:23:52Z)
• AJAR: Adaptive Jailbreak Architecture for Red-teaming [1.356919241968803]
  AJAR is a proof-of-concept framework designed to bridge the gap between "red-teaming" and "action security"<n>AJAR decouples adversarial logic from the execution loop, encapsulating state-of-the-art algorithms like X-Teaming as standardized, plug-and-play services.<n> AJAR is open-sourced to facilitate the standardized, environment-aware evaluation of this emerging attack surface.
  arXiv  Detail & Related papers  (2026-01-16T03:30:40Z)
• ReasAlign: Reasoning Enhanced Safety Alignment against Prompt Injection Attack [52.17935054046577]
  We present ReasAlign, a model-level solution to improve safety alignment against indirect prompt injection attacks.<n>ReasAlign incorporates structured reasoning steps to analyze user queries, detect conflicting instructions, and preserve the continuity of the user's intended tasks.
  arXiv  Detail & Related papers  (2026-01-15T08:23:38Z)
• StriderSPD: Structure-Guided Joint Representation Learning for Binary Security Patch Detection [22.120085662911194]
  Security Patch Detection (SPD) comes to protect software assets.<n>Most SPD studies have targeted Open-Source Software (OSS), yet a large portion of real-world software is closed-source.<n>We propose textbftextitStriderSPD, a framework of binary code that integrates a graph branch into a large language model.
  arXiv  Detail & Related papers  (2026-01-09T12:55:29Z)
• SeCodePLT: A Unified Platform for Evaluating the Security of Code GenAI [58.29510889419971]
  Existing benchmarks for evaluating the security risks and capabilities of code-generating large language models (LLMs) face several key limitations.<n>We introduce a general and scalable benchmark construction framework that begins with manually validated, high-quality seed examples and expands them via targeted mutations.<n>Applying this framework to Python, C/C++, and Java, we build SeCodePLT, a dataset of more than 5.9k samples spanning 44 CWE-based risk categories and three security capabilities.
  arXiv  Detail & Related papers  (2024-10-14T21:17:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.