An Executable Benchmarking Suite for Tool-Using Agents

• URL: http://arxiv.org/abs/2605.11030v1
• Date: Sun, 10 May 2026 21:24:53 GMT
• Title: An Executable Benchmarking Suite for Tool-Using Agents
• Abstract summary: The suite connects WebArena Verified, a SWE-Gym slice with SWE-bench-compatible verification, and MiniWoB++ through common workload adapters.<n>The admitted evidence records latency, invalid-action behavior, patch-generation cost, verifier metadata, replay bindings, and provenance under one auditable contract.
• Score: 3.5322131912413908
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Closed-loop tool-using agents are increasingly evaluated in executable web, code, and micro-task environments, but benchmark reports often conflate workloads, action-generating drivers, and the evidence admitted for systems-facing claims. We present an executable benchmarking suite that makes these objects explicit under a shared evidence-admission contract. The suite connects WebArena Verified, a SWE-Gym slice with SWE-bench-compatible verification, and MiniWoB++ through common workload adapters, task manifests, event schemas, replay/freeze policy, declared drivers, and reporting pipelines. In the canonical release, the gate separates paper-facing evidence from preflight, fixture, smoke, and diagnostic rows while preserving non-admitted artifacts for audit and onboarding. The admitted evidence records latency, invalid-action behavior, patch-generation cost, verifier metadata, replay bindings, and provenance under one auditable contract. The gate is decision-relevant rather than merely clerical: in a separate WebArena Verified controller study, clean-baseline and medium live-stressed evaluation select different fixed controller variants under the same workload and admission contract. The release is scoped as a benchmarking suite and admitted evidence, not a new agent policy, model leaderboard, backend comparison, or autonomous SWE-bench solver.

Related papers

• Disagree to Explore, Agree to Commit: Routing-Guided Test-Time Scaling for Software Agents [60.650808962786364]
  We study whether native MoE router traces can guide steering and selection without an external judge or selection-time test execution.<n>Our analysis shows that routing provides a robust behavioral role signal; token-granular readouts and decision-matched comparison sets turn it into effective control.<n>We evaluate on SWE-bench Verified using open-weight sparse MoE agents across scales and reasoning-effort settings.
  arXiv  Detail & Related papers  (2026-08-23T03:07:03Z)
• Independent Patch Verification for Coding Agents with a Bidirectional Reconstruct-and-Verify Framework [49.16055123488827]
  coding agents powered by large language models can now generate code patches directly from bug reports.<n>No mechanism independently verifies whether it truly resolves the reported problem.<n>We propose RETRACE, a training-free post-generation verification framework.
  arXiv  Detail & Related papers  (2026-08-09T22:59:13Z)
• From Trajectories to Evidence: Auditable Experimental Records for Industrial Research Agents [47.678562263387214]
  Research agents increasingly conduct multi-round machine-learning experiments in industrial recommendation settings.<n>Yet a completed trajectory is not automatically evidence: generated artifacts may be unsupported or incomplete, executed rounds may be invalid or confounded, and later modifications may obscure earlier findings.<n>We introduce an evidence-grounded framework that couples bounded verification of consequential artifacts with post-execution claim qualification.
  arXiv  Detail & Related papers  (2026-08-05T13:37:38Z)
• Agent Security Needs Redefinition through a Holistic Framework [46.88917910966891]
  Defenses ask whether an instruction looks malicious. Benchmarks ask whether an agent performs a harmful sounding action.<n>We operationalize contextual security through four properties that must hold jointly and be evaluated continuously across the agent's trajectory.
  arXiv  Detail & Related papers  (2026-07-24T06:43:09Z)
• Tencent WorkBuddy Bench: A Multi-Domain Coding-Agent Benchmark with Contamination-Resistant Task Construction [57.138342889101345]
  Tencent WorkBuddy Bench is a multi-domain evaluation suite for coding agents.<n>This report documents its construction methodology, scoring protocol, and a cross-model leaderboard.
  arXiv  Detail & Related papers  (2026-07-23T04:34:06Z)
• PhyAgentOS: A Self-Evolving Operating System for Embodied Agents with Decoupled Cognitive Planning and Physical Execution [49.776611937968]
  We present PhyAgentOS, a foundation delivering scheduling, verification, memory, benchmarking, and safety as system-level services.<n>Its Session-Centered treats a session, not an action, as the minimum unit of scheduling, compatibility, supervised execution, evidence collection, and acceptance.<n>SessionVerifier distinguishes execution termination from semantic task completion via evidence-grounded verdicts of success, failure, or replan.<n> Benchmarking reuses the deployment session and verification path, so results trace to real execution.
  arXiv  Detail & Related papers  (2026-07-18T04:46:53Z)
• CAVA: Canonical Action Verification and Attestation for Runtime Governance of Agentic AI Systems [0.6526824510982799]
  This paper presents a runtime-semantics layer for converting heterogeneous agent activity into canonical runtime objects.<n>The contribution is a systems formulation of action-level canonicalization and policy-addressable semantic patterns as a necessary substrate for deployer-side AI governance.
  arXiv  Detail & Related papers  (2026-07-15T11:34:34Z)
• From Tool Connection to Execution Control: Benchmarking Security Invariants in MCP-Style Agent Runtimes [8.419155861590548]
  As agents move from connection to execution, security decisions often remain split across clients, servers, prompts, approval dialogs, deployments, and logs.<n>This paper asks whether a runtime can make execution-layer invariants explicit and testable while preserving MCP-like ecosystems.<n>We implement these invariants in HCP, a Handle-Capability Protocol reference runtime for MCP-style agent execution.
  arXiv  Detail & Related papers  (2026-06-27T20:17:43Z)
• Proof-Carrying Agent Actions: Model-Agnostic Runtime Governance for Heterogeneous Agent Systems [0.6526824510982799]
  This paper presents Proof-Carrying Agent Actions (PCAA), a runtime-neutral governance model centered on an action certificate.<n>PCAA organizes control around five checkpoints: pre-action admissibility, action open, assumption capture, approval, and outcome closure.<n>We study the model through a reference implementation in a heterogeneous agent control plane and a disclosure-bounded evaluation protocol.
  arXiv  Detail & Related papers  (2026-06-02T18:10:35Z)
• Can Agent Benchmarks Support Their Scores? Evidence-Supported Bounds for Interactive-Agent Evaluation [8.049844623361725]
  Interactive agent benchmarks map an agent run to a binary outcome through outcome checks.<n>These checks rely on surface level signals or fail to capture the agent's actual action path.<n>We introduce an outcome evidence reporting layer for existing benchmarks.
  arXiv  Detail & Related papers  (2026-05-11T12:20:15Z)
• SARC: A Governance-by-Architecture Framework for Agentic AI Systems [0.0]
  Agentic AI systems increasingly act through tools, sub-agents, and external services, but governance controls are still commonly attached to prompts, dashboards, or post-hoc documentation.<n>We introduce SARC, a runtime governance architecture for tool-using agents that treats constraints as first-class specification objects alongside state, action space, and reward.<n>SARC executes zero hard-constraint violations under exact predicates; its declared PAA throttling response reduces soft-window overages by 89.5% relative to policy-as-code-only.
  arXiv  Detail & Related papers  (2026-05-08T13:34:36Z)
• Test Before You Deploy: Governing Updates in the LLM Supply Chain [3.6704226968275253]
  Large Language Models (LLMs) are increasingly used as core dependencies in software systems.<n>LLMs evolve continuously through provider-side updates without explicit version changes.<n>This paper proposes a deployment-side governance framework based on three components.
  arXiv  Detail & Related papers  (2026-04-30T12:32:13Z)
• IMPACT-CYCLE: A Contract-Based Multi-Agent System for Claim-Level Supervisory Correction of Long-Video Semantic Memory [73.22944697933603]
  Existing pipelines produce opaque, end-to-end outputs that expose no intermediate state for inspection.<n>We present IMPACT-Cycle, a supervisory multi-agent system that reformulates long-video understanding as a multimodal iterative claim-level maintenance.
  arXiv  Detail & Related papers  (2026-04-22T03:03:33Z)
• AJ-Bench: Benchmarking Agent-as-a-Judge for Environment-Aware Evaluation [71.49152943451328]
  We introduce a benchmark AJ-Bench to evaluate Agent-as-a-Judge across three domains-search, data systems, and graphical user interfaces-comprising 155 tasks and 516 annotated trajectories.<n>Experiments demonstrate consistent performance gains over LLM-as-a-Judge baselines, while also revealing substantial open challenges in agent-based verification.
  arXiv  Detail & Related papers  (2026-04-20T13:23:38Z)
• A Trace-Based Assurance Framework for Agentic AI Orchestration: Contracts, Testing, and Governance [0.22940141855172028]
  This paper presents an assurance framework for Agentic AI systems using Large Language Models (LLMs)<n>Executions are instrumented as Message-Action Traces (MAT) with explicit step and trace contracts.<n>The framework includes stress testing, formulated as a budgeted counterexample search over bounded perturbations.
  arXiv  Detail & Related papers  (2026-03-18T10:23:48Z)
• DEP: A Decentralized Large Language Model Evaluation Protocol [51.3646001384887]
  Decentralized Evaluation Protocol (DEP) is a decentralized yet unified and standardized evaluation framework.<n>By decoupling users, LLMs, and benchmarks, DEP enables modular, plug-and-play evaluation.<n>We develop DEP Toolkit, a protocol-compatible toolkit that supports features such as breakpoint resume, concurrent requests, and congestion control.
  arXiv  Detail & Related papers  (2026-03-01T16:10:16Z)
• OpenPort Protocol: A Security Governance Specification for AI Agent Tool Access [11.289770127178882]
  This paper introduces OpenPort Protocol, a governance-first specification for exposing application tools through a secure server-side gateway.<n>OpenPort defines authorization-dependent discovery, stable response envelopes with machine-actionable texttagent.* reason codes, and an authorization model combining integration credentials, scoped permissions, and ABAC-style policy constraints.<n>For write operations, OpenPort specifies a risk-gated lifecycle that defaults to draft creation and human review, supports time-bounded auto-execution under explicit policy, and enforces high-risk safeguards including preflight impact binding and idempotency
  arXiv  Detail & Related papers  (2026-02-22T05:16:40Z)
• ImpossibleBench: Measuring LLMs' Propensity of Exploiting Test Cases [58.411135609139855]
  "Shortcuts" to complete tasks pose significant risks for reliable assessment and deployment of large language models.<n>We introduce ImpossibleBench, a benchmark framework that measures LLM agents' propensity to exploit test cases.<n>As a practical framework, ImpossibleBench is not just an evaluation but a versatile tool.
  arXiv  Detail & Related papers  (2025-10-23T06:58:32Z)
• DRIFT: Dynamic Rule-Based Defense with Injection Isolation for Securing LLM Agents [52.92354372596197]
  Large Language Models (LLMs) are increasingly central to agentic systems due to their strong reasoning and planning capabilities.<n>This interaction also introduces the risk of prompt injection attacks, where malicious inputs from external sources can mislead the agent's behavior.<n>We propose a Dynamic Rule-based Isolation Framework for Trustworthy agentic systems, which enforces both control and data-level constraints.
  arXiv  Detail & Related papers  (2025-06-13T05:01:09Z)
• CANTXSec: A Deterministic Intrusion Detection and Prevention System for CAN Bus Monitoring ECU Activations [53.036288487863786]
  We propose CANTXSec, the first deterministic Intrusion Detection and Prevention system based on physical ECU activations.<n>It detects and prevents classical attacks in the CAN bus, while detecting advanced attacks that have been less investigated in the literature.<n>We prove the effectiveness of our solution on a physical testbed, where we achieve 100% detection accuracy in both classes of attacks while preventing 100% of FIAs.
  arXiv  Detail & Related papers  (2025-05-14T13:37:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.