Claw-Eval-Live: A Live Agent Benchmark for Evolving Real-World Workflows

• URL: http://arxiv.org/abs/2604.28139v2
• Date: Fri, 01 May 2026 09:39:37 GMT
• Title: Claw-Eval-Live: A Live Agent Benchmark for Evolving Real-World Workflows
• Abstract summary: We introduce Claw-Eval-Live, a live benchmark for workflow agents.<n>Each release is constructed from public workflow-demand signals.<n>Claw-Eval-Live records execution traces, audit logs, service state, and post-run workspace artifacts.
• Score: 67.92316850084575
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: LLM agents are expected to complete end-to-end units of work across software tools, business services, and local workspaces. Yet many agent benchmarks freeze a curated task set at release time and grade mainly the final response, making it difficult to evaluate agents against evolving workflow demand or verify whether a task was executed. We introduce Claw-Eval-Live, a live benchmark for workflow agents that separates a refreshable signal layer, updated across releases from public workflow-demand signals, from a reproducible, time-stamped release snapshot. Each release is constructed from public workflow-demand signals, with ClawHub Top-500 skills used in the current release, and materialized as controlled tasks with fixed fixtures, services, workspaces, and graders. For grading, Claw-Eval-Live records execution traces, audit logs, service state, and post-run workspace artifacts, using deterministic checks when evidence is sufficient and structured LLM judging only for semantic dimensions. The release contains 105 tasks spanning controlled business services and local workspace repair, and evaluates 13 frontier models under a shared public pass rule. Experiments reveal that reliable workflow automation remains far from solved: the leading model passes only 66.7% of tasks and no model reaches 70%. Failures are structured by task family and execution surface, with HR, management, and multi-system business workflows as persistent bottlenecks and local workspace repair comparatively easier but unsaturated. Leaderboard rank alone is insufficient because models with similar pass rates can diverge in overall completion, and task-level discrimination concentrates in a middle band of tasks. Claw-Eval-Live suggests that workflow-agent evaluation should be grounded twice, in fresh external demand and in verifiable agent action.

Related papers

• Inducing Task Models from Computer-Use Traces [50.6293197430374]
  Task Model Induction (TMI) discovers latent tasks in an unconstrained trace and disentangling concurrent activity.<n>TMI reconstructs 74.9% of the observed execution steps, far more than the strongest workflow induction baseline.<n> skills derived from TMI's task models improve held-out task accuracy by 30.0% over the strongest baseline.
  arXiv  Detail & Related papers  (2026-08-20T17:57:00Z)
• ClawArena-Team: Benchmarking Subagent Orchestration and Dynamic Workflows in Language-Model Agents [61.184954205350984]
  ClawArena-Team is a benchmark of 41 multi-turn, multimodal, multi-agent scenarios spanning 258 evaluation rounds and 72 staged updates.<n>All scoring is execution-based with no LLM judge: an overall score -- the Subagent-Management Score (SMS) -- multiplies task correctness by a least-privilege and modality-routing factor.
  arXiv  Detail & Related papers  (2026-06-30T06:06:08Z)
• SaaS-Bench: Can Computer-Use Agents Leverage Real-World SaaS to Solve Professional Workflows? [29.5539656241093]
  Computer-Using Agents (CUAs) are rapidly extending large language models (LLMs) beyond text-based reasoning toward action execution in more complex environments.<n>Existing benchmarks often rely on simplified settings, isolated tasks, or short-horizon interactions.<n>We introduce a benchmark built on 23 deployable systems across six professional domains, containing 106 tasks grounded in realistic work scenarios.<n>Experiments show that representative LLM-based agents struggle on-Bench, with even the strongest model completing fewer than 4% of tasks end-to-end.
  arXiv  Detail & Related papers  (2026-05-15T09:35:15Z)
• GraphFlow: An Architecture for Formally Verifiable Visual Workflows Enabling Reliable Agentic AI Automation [0.0]
  GraphFlow is a visual workflow system designed to improve the reliability of agentic AI automation in mission-critical processes.<n>Existing workflow platforms provide durable execution and observability but offer few semantic correctness guarantees.<n>A year-long pilot across three clinical sites executed 8,728 cohort-enrolled workflow runs with a 97.08% completion rate.
  arXiv  Detail & Related papers  (2026-05-14T15:33:05Z)
• ClawMark: A Living-World Benchmark for Multi-Turn, Multi-Day, Multimodal Coworker Agents [77.73176261673471]
  We introduce bench, a benchmark for coworker agents built around multi-turn multi-day tasks.<n>The current release contains 100 tasks across 13 professional scenarios, executed against five stateful sandboxed services.<n>The strongest model reaches 75.8 weighted score, but the best strict Task Success is only 20.0%, indicating that partial progress is common.
  arXiv  Detail & Related papers  (2026-04-26T16:05:02Z)
• FlowEvo: Self-Evolving Agents through the Co-Evolution of Workflows and Executable Skills [14.679874305674156]
  FlowEvo is a training-free framework that compiles successful traces into reusable skill records.<n>FlowEvo is organized around three coupled mechanisms: (1)workflow-to-skill compilation; (2)skill-to-workflow feedback; and (3)skill curation.
  arXiv  Detail & Related papers  (2026-04-18T06:52:42Z)
• Beyond Rule-Based Workflows: An Information-Flow-Orchestrated Multi-Agents Paradigm via Agent-to-Agent Communication from CORAL [0.15199492741752027]
  We propose an Information-Flow-Orchestrated Multi-Agent Paradigm via Agent-to-Agent (A2A) Communication.<n>We evaluate our approach on the general-purpose benchmark GAIA, using the representative workflow-based MAS as the baseline.<n>Our method achieves 63.64% accuracy, outperforming OWL's 55.15% by 8.49 percentage points with comparable token consumption.
  arXiv  Detail & Related papers  (2026-01-14T21:35:51Z)
• WISE-Flow: Workflow-Induced Structured Experience for Self-Evolving Conversational Service Agents [12.014029662322152]
  Large language model (LLM)-based agents are widely deployed in user-facing services but remain error-prone in new tasks.<n>We propose WISE-Flow, a feasibility-centric framework that converts historical service interactions into reusable procedural experience.
  arXiv  Detail & Related papers  (2026-01-13T02:43:41Z)
• SCUBA: Salesforce Computer Use Benchmark [63.66753028386581]
  SCUBA is a benchmark designed to evaluate computer-use agents on customer relationship management ( CRM) within the Salesforce platform.<n> SCUBA contains 300 task instances derived from real user interviews, spanning three primary personas, platform administrators, sales representatives, and service agents.<n>We benchmark a diverse set of agents under both zero-shot and demonstration-augmented settings.
  arXiv  Detail & Related papers  (2025-09-30T16:48:49Z)
• Benchmarking Agentic Workflow Generation [80.74757493266057]
  We introduce WorfBench, a unified workflow generation benchmark with multi-faceted scenarios and intricate graph workflow structures.<n>We also present WorfEval, a systemic evaluation protocol utilizing subsequence and subgraph matching algorithms.<n>We observe that the generated can enhance downstream tasks, enabling them to achieve superior performance with less time during inference.
  arXiv  Detail & Related papers  (2024-10-10T12:41:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.