ClawTrack: Towards Trace-Level Evaluation and Improvement of Real-World Autonomous Agents

• URL: http://arxiv.org/abs/2607.28037v1
• Date: Thu, 30 Jul 2026 11:18:47 GMT
• Title: ClawTrack: Towards Trace-Level Evaluation and Improvement of Real-World Autonomous Agents
• Abstract summary: We present ClawTrack, a dual-assessment benchmark that simultaneously measures what an agent achieves and how it achieves it.<n>ClawTrack comprises 320 tasks across 8 domains with 25+ deterministic mock services.<n>A Process Grader scores each reasoning turn along four dimensions (goal alignment, efficiency, information utilization, and result verification) anchored by 12,541 task-specific items.
• Score: 23.840537431318296
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: As LLM-based agents are deployed in complex, multi-step workflows, a critical evaluation gap has emerged: most existing benchmarks judge only final outcomes, unable to distinguish reliable reasoning from lucky success or attribute failures to specific process deficiencies, hindering attribution in long-horizon tasks. In this work, we present ClawTrack, a dual-assessment benchmark that simultaneously measures what an agent achieves (Task Score) and how it achieves it (Process Score). ClawTrack comprises 320 tasks across 8 domains with 25+ deterministic mock services. A Process Grader scores each reasoning turn along four dimensions (goal alignment, efficiency, information utilization, and result verification), anchored by 12,541 task-specific rubric items. Evaluating 21 models over 16,000+ trials, we find that: (1) process scores effectively attribute success and failure to specific reasoning dimensions, filtering lucky passes invisible to outcome-only evaluation; (2) the four dimensions are complementary, with result verification as the systematic bottleneck; (3) the framework is robust to evaluator choice across different judge LLMs; and (4) process-based trajectory filtering yields consistent post-training improvements across model scales.

Related papers

• CalibForge: Adversarial Solver Calibration for Scaling Learnable Terminal Tasks [87.1086886770395]
  CalibForge is an autonomous terminal-task synthesis system.<n>It uses verified solver behavior to revise candidate tasks through adversarial solver calibration.<n>Using CalibForge, we construct 5,431 terminal tasks.
  arXiv  Detail & Related papers  (2026-08-06T17:53:18Z)
• Operationalising Multi-Dimensional Evaluation for Conversational Agents: A Scalable, Governed Pipeline with Selective Re-evaluation and Model Benchmarking [0.0]
  We present GenAI Evaluation, a governed, configuration-driven pipeline for large-scale evaluation of retail conversational systems.<n>The framework evaluates helpfulness, truthfulness, clarity, tone alignment, and translation-specific dimensions.<n>The pipeline achieved a macro F1 score of 0.93 and 89% human-acceptability accuracy for translation.
  arXiv  Detail & Related papers  (2026-07-13T19:01:56Z)
• Claw-Eval: Toward Trustworthy Evaluation of Autonomous Agents [66.97968363332465]
  We introduce Claw-Eval, an end-to-end evaluation suite addressing all three gaps in agent benchmarks.<n>Claw-Eval comprises 300 human-verified tasks spanning 9 categories across three groups.<n>Every agent action is recorded through three independent evidence channels.
  arXiv  Detail & Related papers  (2026-04-07T17:43:18Z)
• Beyond Binary Correctness: Scaling Evaluation of Long-Horizon Agents on Subjective Enterprise Tasks [4.09533297040075]
  Large language models excel on objectively verifiable tasks such as math and programming, where evaluation reduces to unit tests or a single correct answer.<n>We introduce LH-Bench, a three-pillar evaluation design that moves beyond binary correctness to score autonomous, long-horizon execution on subjective enterprise tasks.
  arXiv  Detail & Related papers  (2026-03-24T03:16:32Z)
• Verified Critical Step Optimization for LLM Agents [67.05296684575445]
  Critical Step Optimization focuses preference learning on verified critical steps.<n>Method starts from failed policy trajectories rather than expert demonstrations.<n>Experiments on GAIA-Text-103 and XBench-DeepSearch show that CSO achieves 37% and 26% relative improvement over the SFT baseline.
  arXiv  Detail & Related papers  (2026-02-03T11:41:02Z)
• Model Specific Task Similarity for Vision Language Model Selection via Layer Conductance [92.72779885657373]
  We propose a framework that grounds model selection in the internal functional dynamics of the visual encoder.<n>Our approach represents each task via layer wise conductance and derives a target-conditioned block importance distribution through entropy regularized alignment.<n>Building on this, we introduce Directional Conductance Divergence (DCD), an asymmetric metric that quantifies how effectively a source task covers the target's salient functional blocks.
  arXiv  Detail & Related papers  (2026-02-01T17:29:43Z)
• CoT4Det: A Chain-of-Thought Framework for Perception-Oriented Vision-Language Tasks [53.88194225946438]
  Chain-of-Thought for Detection (CoT4Det) is a simple but efficient strategy that reformulates perception tasks into three interpretable steps.<n>We show that CoT4Det significantly improves perception performance without compromising general vision language capabilities.
  arXiv  Detail & Related papers  (2025-12-07T05:26:30Z)
• DrawingBench: Evaluating Spatial Reasoning and UI Interaction Capabilities of Large Language Models through Mouse-Based Drawing Tasks [10.977990951788422]
  DrawingBench is a verification framework for evaluating the trustworthiness of agentic LLMs.<n>Our framework comprises 250 diverse prompts across 20 categories and 4 difficulty levels.<n>We evaluate four state-of-the-art LLMs across 1,000 tests.
  arXiv  Detail & Related papers  (2025-12-01T01:18:21Z)
• TrajSelector: Harnessing Latent Representations for Efficient and Effective Best-of-N in Large Reasoning Model [21.82904448561646]
  Large language models (LLMs) have shown remarkable progress in complex reasoning tasks.<n>Best-of-N selection paradigm yields scalable performance improvements by selecting from multiple independently generated reasoning trajectories.<n>We introduce TrajSelector, an efficient and effective Best-of-N framework that exploit the hidden states in the sampler LLM for process-level scoring.
  arXiv  Detail & Related papers  (2025-10-18T11:01:39Z)
• Eigen-1: Adaptive Multi-Agent Refinement with Monitor-Based RAG for Scientific Reasoning [53.45095336430027]
  We develop a unified framework that combines implicit retrieval and structured collaboration.<n>On Humanity's Last Exam (HLE) Bio/Chem Gold, our framework achieves 48.3% accuracy.<n>Results on SuperGPQA and TRQA confirm robustness across domains.
  arXiv  Detail & Related papers  (2025-09-25T14:05:55Z)
• SPARE: Single-Pass Annotation with Reference-Guided Evaluation for Automatic Process Supervision and Reward Modelling [58.05959902776133]
  We introduce Single-Pass.<n>with Reference-Guided Evaluation (SPARE), a novel structured framework that enables efficient per-step annotation.<n>We demonstrate SPARE's effectiveness across four diverse datasets spanning mathematical reasoning (GSM8K, MATH), multi-hop question answering (MuSiQue-Ans), and spatial reasoning (SpaRP)<n>On ProcessBench, SPARE demonstrates data-efficient out-of-distribution generalization, using only $sim$16% of training samples compared to human-labeled and other synthetically trained baselines.
  arXiv  Detail & Related papers  (2025-06-18T14:37:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.