STAGE-Claw: Automated State-based Agent Benchmarking for Realistic Scenarios

• URL: http://arxiv.org/abs/2606.10394v1
• Date: Tue, 09 Jun 2026 04:16:35 GMT
• Title: STAGE-Claw: Automated State-based Agent Benchmarking for Realistic Scenarios
• Abstract summary: This paper introduces STAGE-Claw, an automated framework for building and evaluating realistic personal-agent scenarios.<n>Using STAGE-Claw, this paper creates a benchmark with 40 challenging real scenario agent tasks, evaluates 11 frontier models, and analyzes their task scores, costs, tool-call reliability, and common failure patterns.
• Score: 38.892583102842956
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models are increasingly used to power personal agents for everyday applications, but evaluating these agents remains a challenge. Existing benchmarks still rely on sandboxed artifacts, static task design, and coarse scoring, which hinder scalability and limit progress toward reliable personal-agent evaluation. This paper introduces STAGE-Claw, an automated framework for building and evaluating realistic personal-agent scenarios in state-based personal-computing environments. Given a task hint, STAGE-Claw automatically creates and validates a realistic benchmark task with its environment, task prompts, ground truth, and related verification programs. Agents are then evaluated in realistic operating environments, where performance is measured by the correctness of the final system state rather than only the textual response. Using STAGE-Claw, this paper creates a benchmark with 40 challenging real scenario agent tasks, evaluates 11 frontier models, and analyzes their task scores, costs, tool-call reliability, and common failure patterns. Overall, STAGE-Claw offers a scalable, state-based way to evaluate agents in realistic user scenarios.

Related papers

• DuMateBench: Evaluating Autonomous Agents in Complex Real-World Workflows [69.12339622053588]
  We introduce DuMateBench, a real-session benchmark reconstructed from anonymized and privacy-screened user sessions.<n>The resulting benchmark comprises 200 tasks spanning 8 broad scenarios and 17 fine-grained capability categories.<n>We execute these tasks in isolated Docker containers injected with three forms of real-world environmental complexity: Insufficient, Unstable, and Noisy.
  arXiv  Detail & Related papers  (2026-08-27T02:36:23Z)
• AgentGym2: Benchmarking Large Language Model Agents in De-Idealized Real-World Environments [84.8170304874972]
  We present AgentGym2, a new evaluation framework with task instances grounded in real-world working demands.<n>It measures agents' ability to execute end-to-end procedures, discover tools via exploration, compose tools for unseen tasks, and remain robust to noisy and underspecified information.
  arXiv  Detail & Related papers  (2026-07-06T14:56:51Z)
• SPADE-Bench: Evaluating Spontaneous Strategic Deception in Agents via Plan-Action Divergence [30.99083906950254]
  We introduce SPADE-Bench, a benchmark designed to evaluate spontaneous plan-action divergence.<n>Unlike prior deception benchmarks, SPADE-Bench simultaneously integrates actual tool execution and controlled pressure scenarios.<n> Experiments confirm that agent deception is a genuine and pressing issue in tool-use contexts.
  arXiv  Detail & Related papers  (2026-06-01T15:28:34Z)
• An Agentic Retrieval Framework for Autonomous Context-Aware Data Quality Assessment [0.0]
  We propose a unified agentic-retrieval framework for autonomous context-aware data quality assessment.<n>The framework interprets natural-language descriptions of intended data usage, derives context-aware assessment strategies, and generates executable validation logic.<n>We implement the proposed framework as an end-to-end prototype and evaluate it across multiple usage scenarios applied to the same dataset.
  arXiv  Detail & Related papers  (2026-05-15T11:24:53Z)
• ClawForge: Generating Executable Interactive Benchmarks for Command-Line Agents [59.626170560327274]
  We present textbfClawForge, a generator-backed benchmark framework for executable command-line categories under state conflict.<n>We instantiate this framework as the ClawForge-Bench (17 scenarios, 6 ability categories)
  arXiv  Detail & Related papers  (2026-05-13T21:34:08Z)
• CUAAudit: Meta-Evaluation of Vision-Language Models as Auditors of Autonomous Computer-Use Agents [15.119045051735633]
  Computer-Use Agents (CUAs) are emerging as a new paradigm in human-computer interaction, enabling autonomous execution of tasks in desktop environment by perceiving high-level natural-language instructions.<n>Existing evaluation pipelines rely on static benchmarks, rule-based success checks, or manual inspection, which are brittle, costly, and poorly aligned with real-world usage.<n>We study Vision-Language Models (VLMs) as autonomous auditors for assessing CUA task completion directly from observable interactions.<n>We find that while state-of-the-art VLMs achieve strong accuracy and calibration, all auditors exhibit notable performance degradation in more complex
  arXiv  Detail & Related papers  (2026-03-11T09:28:41Z)
• Process-Level Trajectory Evaluation for Environment Configuration in Software Engineering Agents [71.85020581835042]
  Large language model-based agents show promise for software engineering, but environment configuration remains a bottleneck.<n>Existing benchmarks assess only end-to-end build/test success, obscuring where and why agents succeed or fail.<n>We introduce Enconda-bench, which provides process-level trajectory assessment of fine-grained agent capabilities during environment setup-planning.
  arXiv  Detail & Related papers  (2025-10-29T16:59:07Z)
• Shell or Nothing: Real-World Benchmarks and Memory-Activated Agents for Automated Penetration Testing [23.554239007767276]
  We introduce the first real-world, agent-oriented pentesting benchmark, TermiBench.<n>We propose TermiAgent, a multi-agent penetration testing framework.<n>In evaluations, our work outperforms state-of-the-art agents, exhibiting stronger penetration testing capability.
  arXiv  Detail & Related papers  (2025-09-11T07:30:44Z)
• FieldWorkArena: Agentic AI Benchmark for Real Field Work Tasks [52.47895046206854]
  FieldWorkArena is a benchmark for agentic AI targeting real-world field work.<n>This paper defines a new action space that agentic AI should possess for real world work environment benchmarks.
  arXiv  Detail & Related papers  (2025-05-26T08:21:46Z)
• SOPBench: Evaluating Language Agents at Following Standard Operating Procedures and Constraints [59.645885492637845]
  SOPBench is an evaluation pipeline that transforms each service-specific SOP code program into a directed graph of executable functions.<n>Our approach transforms each service-specific SOP code program into a directed graph of executable functions and requires agents to call these functions based on natural language SOP descriptions.<n>We evaluate 18 leading models, and results show the task is challenging even for top-tier models.
  arXiv  Detail & Related papers  (2025-03-11T17:53:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.