Claw-SWE-Bench: A Benchmark for Evaluating OpenClaw-style Agent Harnesses on Coding Tasks

• URL: http://arxiv.org/abs/2606.12344v1
• Date: Wed, 10 Jun 2026 17:16:23 GMT
• Title: Claw-SWE-Bench: A Benchmark for Evaluating OpenClaw-style Agent Harnesses on Coding Tasks
• Abstract summary: General-purpose agents such as OpenClaw are increasingly used as autonomous tool users.<n>We introduce Claw-SWE-Bench, a multilingual SWE-bench-style benchmark and adapter protocol.<n>Claw-SWE-Bench treats harness and cost accounting as first-class axes of SWE-style coding-agent evaluation.
• Score: 75.92297551160692
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: General-purpose agents such as OpenClaw are increasingly used as autonomous tool users, but their coding ability is difficult to measure under SWE-bench: a generic agent does not by itself satisfy the clean Docker workspace, patch, and prediction contract required for scoring. We introduce Claw-SWE-Bench, a multilingual SWE-bench-style benchmark and adapter protocol that makes heterogeneous agent harnesses, or claws, comparable under fair settings including a fixed prompt, runtime budget, workspace contract, patch extraction procedure, and evaluator. The full benchmark contains 350 GitHub issue-resolution instances across 8 languages and 43 repositories, drawn from SWE-bench-Multilingual and SWE-bench-Verified-Mini after future-commit cleanup. We also release Claw-SWE-Bench Lite for faster validation, which is an 80-instance subset selected by a cost-aware, rank-aware procedure over 17 calibration columns. On the full benchmark, OpenClaw with a minimal direct-diff adapter scores only $19.1\%$ Pass@1, whereas the full adapter reaches $73.4\%$ with the same GLM 5.1 backbone, showing that adapter design is essential for enabling OpenClaw-style harnesses to perform coding tasks effectively. Across an OpenClaw $\times$ nine-model sweep and a five-claw $\times$ two-model sweep, model choice changes Pass@1 by $29.4$ pp and harness choice by $27.4$ pp under fixed models; systems with similar accuracy can differ substantially in total API cost. Claw-SWE-Bench therefore treats harness and cost accounting as first-class axes of SWE-style coding-agent evaluation, providing both a full benchmark and a low-cost reference set for reproducible comparison. The data is available at https://github.com/opensquilla/claw-swe-bench and https://huggingface.co/datasets/TokenRhythm/Claw-SWE-Bench.

Related papers

• SSTQ:Privacy-Preserving Vector Quantization via Subsampled Stochastic TurboQuant [79.24089819400126]
  Subsampled TurboQuant (SSTQ) is a framework that combines overcomplete equal-width tight frames, coordinate subsampling, and privacy-aware quantization.<n>SSTQ achieves mean squared error scaling while using only $lceil log N il + b$ bits per client, where $N = (d)$ is the frame size.<n>We also derive a surrogate privacy-aware codebook objective that reduces the codebook-dependent MSE scaling from $O(4b)$ to $O(2b)$.
  arXiv  Detail & Related papers  (2026-08-05T17:51:25Z)
• When Does Restricting a Coding Agent to execute_code Help? A Regime $\ imes$ Agent-Design Ablation [20.672156968511587]
  We run a comparison of three-arm ablation (baseline / bash_only / code_only) on synthetic computation tasks and SWE-bench Mini modification tasks.<n>Across the four resulting (regime, agent) cells, restricting the agent to a single execute_code MCP tool is cheaper than -- or statistically tied with -- its cheapest tool-rich rival.<n>The lone exception is SWE-bench/Claude, where code_only is directionally costlier.
  arXiv  Detail & Related papers  (2026-07-12T04:52:08Z)
• CFAgentBench: A Reproducible Environment and Benchmark for Autonomous Construction-Finance Agents [0.0]
  CFAgentBench is a benchmark for autonomous construction-finance agents.<n>It contains 1,014 machine-gradeable task specifications across 8 domains and 77 families.<n>The strongest agent reaches pass1 = 0.67 but only pass5 = 0.38.
  arXiv  Detail & Related papers  (2026-06-20T11:34:52Z)
• VisualClaw: A Real-Time, Personalized Agent for the Physical World [76.89211120690028]
  We present VisualClaw, a self-evolving multimodal agent built around two principles.<n>First, hybrid encoding reduces deployment cost by filtering less informative streaming frames with a cascaded gate.<n>Second, skill evolution lets the agent learn from failures, producing skill-bank updates that help future questions.
  arXiv  Detail & Related papers  (2026-06-15T06:58:22Z)
• TensorBench: Benchmarking Coding Agents on a Compiler-Based Tensor Framework [0.9012337166501982]
  CohenBench is a benchmark for tasks on an open-source compiler-based framework.<n>It grades each run by applying the agent's patch and running the framework's test suite.<n>We evaluate seven coding agents spanning three frontier model families and one open-weight model.
  arXiv  Detail & Related papers  (2026-06-04T01:42:40Z)
• RRISE: Robust Radius Inference via a Surrogate Estimator [30.521314184750867]
  We introduce RRISE, an RS framework that compresses certification into a single forward pass through a learned surrogate.<n>On CIFAR-100 and Tiny ImageNet, where the only prior offline-surrogate method collapses, RRISE achieves $1.23$ to $1.91times$ higher certified accuracy.
  arXiv  Detail & Related papers  (2026-06-01T20:46:06Z)
• HyDRA: Hybrid Dynamic Routing Architecture for Heterogeneous LLM Pools [2.0320563270126493]
  We present HyDRA, a framework that predicts fine-grained, multi-dimensional capability requirements per query.<n>A shortfall-matching algorithm then selects the cheapest model whose capabilities meet the predicted requirements.<n>The framework is deployed to all users in GitHub Copilot's VS Code Chat auto-mode.
  arXiv  Detail & Related papers  (2026-05-16T18:19:30Z)
• WildClawBench: A Benchmark for Real-World, Long-Horizon Agent Evaluation [88.10947115397971]
  This work presents WildClawBench, a native-runtime benchmark of 60 human-authored, bilingual, multimodal tasks spanning six thematic categories.<n>Each task averages roughly 8 minutes of wall-clock time and over 20 tool calls.<n> Grading is hybrid, combining deterministic rule-based checks, environment-state auditing of side effects, and an LLM/VLM judge for semantic verification.
  arXiv  Detail & Related papers  (2026-05-11T17:49:43Z)
• Correction and Corruption: A Two-Rate View of Error Flow in LLM Protocols [51.56484100374058]
  We propose a paired-outcome measurement interface for auditing a single protocol step on exact-match tasks.<n>For each instance, the interface records a baseline correctness bit and a post-step correctness bit.<n>These rates predict accuracy changes and define a reusable empirical interface testable across seeds, mixtures, and pipelines.
  arXiv  Detail & Related papers  (2026-04-20T13:25:40Z)
• Don't Overthink It: Inter-Rollout Action Agreement as a Free Adaptive-Compute Signal for LLM Agents [0.0]
  Inference-time compute scaling has emerged as a powerful technique for improving the reliability of large language model (LLM) agents.<n>We introduce TrACE, a training-free controller that allocates LLM calls adaptively across agent timesteps by measuring inter-rollout action agreement.
  arXiv  Detail & Related papers  (2026-04-09T15:34:22Z)
• MetaClaw: Just Talk -- An Agent That Meta-Learns and Evolves in the Wild [74.7263562191605]
  Large language model (LLM) agents are increasingly used for complex tasks.<n>Existing methods either store raw trajectories without distilling knowledge, maintain static skill libraries, or require disruptive downtime for retraining.<n>We present MetaClaw, a continual meta-learning framework that jointly evolves a base LLM policy and a library of reusable behavioral skills.
  arXiv  Detail & Related papers  (2026-03-17T22:30:30Z)
• SWE-Bench++: A Framework for the Scalable Generation of Software Engineering Benchmarks from Open-Source Repositories [2.951332247539421]
  We introduce SWE-Bench++, an automated framework that generates repository-level coding tasks from open-source GitHub projects.<n>Unlike synthetic approaches, our pipeline harvests live pull requests to cover both bug fixes and feature requests across 11 languages.<n>Our initial benchmark consists of 11,133 instances from 3,971 repositories across 11 languages.
  arXiv  Detail & Related papers  (2025-12-19T10:16:51Z)
• SWE-Fixer: Training Open-Source LLMs for Effective and Efficient GitHub Issue Resolution [56.9361004704428]
  Large Language Models (LLMs) have demonstrated remarkable proficiency across a variety of complex tasks.<n>SWE-Fixer is a novel open-source framework designed to effectively and efficiently resolve GitHub issues.<n>We assess our approach on the SWE-Bench Lite and Verified benchmarks, achieving competitive performance among open-source models.
  arXiv  Detail & Related papers  (2025-01-09T07:54:24Z)
• Higher-Order Certification for Randomized Smoothing [78.00394805536317]
  We propose a framework to improve the certified safety region for smoothed classifiers. We provide a method to calculate the certified safety region using $0th$-order and $1st$-order information. We also provide a framework that generalizes the calculation for certification using higher-order information.
  arXiv  Detail & Related papers  (2020-10-13T19:35:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.