WildClawBench: A Benchmark for Real-World, Long-Horizon Agent Evaluation

• URL: http://arxiv.org/abs/2605.10912v1
• Date: Mon, 11 May 2026 17:49:43 GMT
• Title: WildClawBench: A Benchmark for Real-World, Long-Horizon Agent Evaluation
• Abstract summary: This work presents WildClawBench, a native-runtime benchmark of 60 human-authored, bilingual, multimodal tasks spanning six thematic categories.<n>Each task averages roughly 8 minutes of wall-clock time and over 20 tool calls.<n> Grading is hybrid, combining deterministic rule-based checks, environment-state auditing of side effects, and an LLM/VLM judge for semantic verification.
• Score: 88.10947115397971
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language and vision-language models increasingly power agents that act on a user's behalf through command-line interface (CLI) harnesses. However, most agent benchmarks still rely on synthetic sandboxes, short-horizon tasks, mock-service APIs, and final-answer checks, leaving open whether agents can complete realistic long-horizon work in the runtimes where they are deployed. This work presents WildClawBench, a native-runtime benchmark of 60 human-authored, bilingual, multimodal tasks spanning six thematic categories. Each task averages roughly 8 minutes of wall-clock time and over 20 tool calls, and runs inside a reproducible Docker container hosting an actual CLI agent harness (OpenClaw, Claude Code, Codex, or Hermes Agent) with access to real tools rather than mock services. Grading is hybrid, combining deterministic rule-based checks, environment-state auditing of side effects, and an LLM/VLM judge for semantic verification. Across 19 frontier models, the best, Claude Opus 4.7, reaches only 62.2% overall under OpenClaw, while every other model stays below 60%, and switching harness alone shifts a single model by up to 18 points. These results show that long-horizon, native-runtime agent evaluation remains a far-from-resolved task for current frontier models. We release the tasks, code, and containerized tooling to support reproducible evaluation.

Related papers

• Agentic Coding in the Wild: Characterizing GitHub Copilot Traces at Production Scale [6.287658270510339]
  AI coding agents like GitHub Copilot, Claude Code, and Codex interleave multi-step LLM inference with tool execution.<n>We present the first production-scale characterization of this workload using sampled GitHub Copilot traces from June 2026.
  arXiv  Detail & Related papers  (2026-07-30T20:51:51Z)
• ClawArena-Team: Benchmarking Subagent Orchestration and Dynamic Workflows in Language-Model Agents [61.184954205350984]
  ClawArena-Team is a benchmark of 41 multi-turn, multimodal, multi-agent scenarios spanning 258 evaluation rounds and 72 staged updates.<n>All scoring is execution-based with no LLM judge: an overall score -- the Subagent-Management Score (SMS) -- multiplies task correctness by a least-privilege and modality-routing factor.
  arXiv  Detail & Related papers  (2026-06-30T06:06:08Z)
• MacAgentBench: Benchmarking AI Agents on Real-World macOS Desktop [32.14321067864185]
  Computer use agents (CUAs) have advanced rapidly in desktop automation, and a growing number of users deploy CUAs for always-on automation.<n>Existing benchmarks, including those for metrics, evaluate agents without framework augmentation and rely on binary evaluation.<n>We present MacAgentBench, a comprehensive agent benchmark comprising 676 tasks across 25 applications.
  arXiv  Detail & Related papers  (2026-06-21T15:32:57Z)
• WeaveBench: A Long-Horizon, Real-World Benchmark for Computer-Use Agents with Hybrid Interfaces [27.11445886768225]
  We introduce WeaveBench, a long-horizon hybrid-interface benchmark with 114 tasks across 8 real-world work domains.<n>We evaluate these tasks on a real Ubuntu desktop inside deployed CLI-agent runtimes, augmented with a minimal desktop-control plugin.<n>Across frontier model-runtime pairings, the best PassRate reaches only 41.2%, showing the benchmark remains far from saturated.
  arXiv  Detail & Related papers  (2026-06-08T12:39:23Z)
• AcademiClaw: When Students Set Challenges for AI Agents [69.70303995121154]
  AcademiClaw is a benchmark of 80 complex, long-horizon tasks sourced directly from university students' real academic disconnects.<n>We show that even the best of models achieves only a 55% pass rate.<n>We hope that AcademiClaw and its open-sourced data and code can serve as a useful resource for the OpenClaw community.
  arXiv  Detail & Related papers  (2026-05-04T14:40:42Z)
• Claw-Eval-Live: A Live Agent Benchmark for Evolving Real-World Workflows [67.92316850084575]
  We introduce Claw-Eval-Live, a live benchmark for workflow agents.<n>Each release is constructed from public workflow-demand signals.<n>Claw-Eval-Live records execution traces, audit logs, service state, and post-run workspace artifacts.
  arXiv  Detail & Related papers  (2026-04-30T17:23:19Z)
• KISS Sorcar: A Stupidly-Simple General-Purpose and Software Engineering AI Assistant [8.273738980475393]
  KISS Sorcar is a general-purpose assistant and integrated development environment built on top of the KISS Agent Framework.<n> KISS Sorcar is a free, open-source Visual Studio Code extension that runs locally and effectively for long-horizon tasks.<n>The entire system was built using itself in 4.5 months, providing a continuous stress test in which any agent-introduced bug immediately impairs its own ability to work.
  arXiv  Detail & Related papers  (2026-04-26T17:59:06Z)
• ClawMark: A Living-World Benchmark for Multi-Turn, Multi-Day, Multimodal Coworker Agents [77.22389710754452]
  We introduce bench, a benchmark for coworker agents built around multi-turn multi-day tasks.<n>The current release contains 100 tasks across 13 professional scenarios, executed against five stateful sandboxed services.<n>The strongest model reaches 75.8 weighted score, but the best strict Task Success is only 20.0%, indicating that partial progress is common.
  arXiv  Detail & Related papers  (2026-04-26T16:05:02Z)
• SWE-EVO: Benchmarking Coding Agents in Long-Horizon Software Evolution Scenarios [6.776894728701934]
  Existing benchmarks for AI coding agents focus on isolated, single-issue tasks such as fixing a bug or implementing a small feature.<n>We introduce SWE-EVO, a benchmark that evaluates agents on a long-horizon software evolution challenge.<n>Tool comprises 48 evolution tasks that require agents to implement multi-step modifications spanning an average of 21 files.
  arXiv  Detail & Related papers  (2025-12-20T19:08:15Z)
• CRAB: Cross-environment Agent Benchmark for Multimodal Language Model Agents [49.68117560675367]
  Crab is the first benchmark framework designed to support cross-environment tasks.<n>Our framework supports multiple devices and can be easily extended to any environment with a Python interface.<n>The experimental results demonstrate that the single agent with GPT-4o achieves the best completion ratio of 38.01%.
  arXiv  Detail & Related papers  (2024-07-01T17:55:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.