An Empirical Study of Automating Agent Evaluation

• URL: http://arxiv.org/abs/2605.11378v1
• Date: Tue, 12 May 2026 01:06:34 GMT
• Title: An Empirical Study of Automating Agent Evaluation
• Abstract summary: We introduce EvalAgent, an AI assistant that automates the end-to-end agent evaluation pipeline.<n>EvalAgent encodes evaluation domain expertise as evaluation skills.<n>We show that EvalAgent produces focused evaluations, improving Eval@1 from 17.5% to 65%, and achieving 79.5% human expert preference over baseline approaches.
• Score: 14.239299198848764
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Agent evaluation requires assessing complex multi-step behaviors involving tool use and intermediate reasoning, making it costly and expertise-intensive. A natural question arises: can frontier coding assistants reliably automate this evaluation process? Our study shows that simply prompting coding assistants is insufficient for this task. Without domain-specific evaluation knowledge, frontier coding assistants achieve only a 30% execution success rate and produce over-engineered evaluations averaging 12+ metrics per agent, indicating that strong coding ability does not automatically translate to reliable agent evaluation. We introduce EvalAgent, an AI assistant that automates the end-to-end agent evaluation pipeline. EvalAgent encodes evaluation domain expertise as evaluation skills (procedural instructions, reusable code and templates, and dynamically retrieved API documentation) that compose into a trace-based pipeline producing complete evaluation artifacts including metrics, executable code, and reports. To systematically assess generated evaluations, we introduce a meta-evaluation framework alongside AgentEvalBench, a benchmark comprising 20 agents, each paired with evaluation requirements and test scenarios. We further propose the Eval@1 metric to measure whether generated evaluation code both executes and yields meaningful results on the first run. Our experiments show that EvalAgent produces focused evaluations, improving Eval@1 from 17.5% to 65%, and achieving 79.5% human expert preference over baseline approaches. Further ablation studies show that evaluation skills are critical for handling complex evaluation: removing them causes Eval@1 to drop significantly from 65% to 30%.

Related papers

• Open Evaluation Agent: Efficient and Promptable Evaluation of Visual Generative Models [74.1961755473754]
  Existing evaluation methods rely on rigid pipelines that overlook specific user needs and provide numerical results without clear explanations.<n>We propose the Evaluation Agent framework, which employs human-like strategies for efficient, dynamic, multi-round evaluations.<n>We show that Evaluation Agent reduces evaluation time to 10% of traditional methods while delivering comparable results.
  arXiv  Detail & Related papers  (2026-08-10T14:42:10Z)
• Training Therapeutic Judges and Multi-Agent Systems for Human-Aligned Mental Health Support [22.031194303408792]
  We introduce a framework that formulates therapeutic response generation as a decision-refinement problem driven by human-aligned evaluation.<n>In Stage I, we introduce TheraJudge, an open-source therapeutic evaluator trained via preference-based optimization on human-annotated data.<n>In Stage II, we introduce TheraAgent, which operationalizes TheraJudge's evaluations through a coordinated refinement process.
  arXiv  Detail & Related papers  (2026-06-29T20:22:25Z)
• The Meta-Agent Challenge: Are Current Agents Capable of Autonomous Agent Development? [80.24951682268332]
  We introduce the Meta-Agent Challenge (MAC), an evaluation framework designed to test the capacity of frontier models for autonomous agent development.<n>To ensure evaluation integrity, this framework is secured by multi-layer defenses against reward hacking.<n>We demonstrate that meta-agents rarely match human-engineered baseline policies, and the few that do are dominated by proprietary frontier models.
  arXiv  Detail & Related papers  (2026-06-03T04:58:17Z)
• Talk, Evaluate, Diagnose: User-aware Agent Evaluation with Automated Error Analysis [3.3237915628874632]
  We argue that effective agent evaluation goes beyond correctness alone, incorporating conversation quality, efficiency and systematic diagnosis of agent errors.<n>We propose new metrics that capture both turn efficiency and intermediate progress of the agent.<n>We show that our TED framework reveals new insights regarding agent performance across models and user expertise levels.
  arXiv  Detail & Related papers  (2026-03-16T16:14:28Z)
• A2Eval: Agentic and Automated Evaluation for Embodied Brain [26.357063836707223]
  Current embodied VLM evaluation relies on static, expert-defined, manually annotated benchmarks.<n>Agentic Automatic Evaluation (A2Eval) is the first agentic framework that automates benchmark curation and evaluation through two collaborative agents.<n> Evaluated across 10 benchmarks and 13 models, A2Eval compresses evaluation suites by 85%, reduces overall computational costs by 77%, and delivers a 4.6x speedup.
  arXiv  Detail & Related papers  (2026-02-02T04:55:27Z)
• Foundational Automatic Evaluators: Scaling Multi-Task Generative Evaluator Training for Reasoning-Centric Domains [97.5573252172065]
  We train a family of Automatic Reasoning Evaluators (FARE) with a simple iterative rejection-sampling supervised finetuning approach.<n>FARE-8B challenges larger specialized RL-trained evaluators and FARE-20B sets the new standard for open-source evaluators.<n>As inference-time rerankers, FARE-20B achieves near-oracle performance on MATH.
  arXiv  Detail & Related papers  (2025-10-20T17:52:06Z)
• Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation [87.47155146067962]
  We provide a standardized evaluation harness that orchestrates parallel evaluations across hundreds of tasks.<n>We conduct three-dimensional analysis spanning models, scaffolds, and benchmarks.<n>Our analysis reveals surprising insights, such as higher reasoning effort reducing accuracy in the majority of runs.
  arXiv  Detail & Related papers  (2025-10-13T22:22:28Z)
• Establishing Best Practices for Building Rigorous Agentic Benchmarks [94.69724201080155]
  We show that many agentic benchmarks have issues in task setup or reward design.<n>Such issues can lead to under- or overestimation of agents' performance by up to 100% in relative terms.<n>We introduce the Agentic Benchmark Checklist (ABC), a set of guidelines that we synthesized from our benchmark-building experience.
  arXiv  Detail & Related papers  (2025-07-03T17:35:31Z)
• AgentRewardBench: Evaluating Automatic Evaluations of Web Agent Trajectories [61.38499597241457]
  We propose AgentRewardBench, the first benchmark to assess the effectiveness of LLM judges for evaluating web agents.<n>Using our benchmark, we evaluate 12 LLM judges and find that no single LLM excels across all benchmarks.<n>We also find that the rule-based evaluation used by common benchmarks tends to underreport the success rate of web agents.
  arXiv  Detail & Related papers  (2025-04-11T19:49:22Z)
• AutoEval: A Practical Framework for Autonomous Evaluation of Mobile Agents [5.995751996623217]
  We propose AutoEval, an evaluation framework which tests mobile agents without any manual effort.<n>Our approach designs a UI state change representation which can be used to automatically generate task reward signals.<n>We also evaluate state-of-the-art mobile agents using our framework, providing insights into their performance and limitations.
  arXiv  Detail & Related papers  (2025-03-04T08:44:30Z)
• Automating the Correctness Assessment of AI-generated Code for Security Contexts [8.009107843106108]
  We propose a fully automated method, named ACCA, to evaluate the correctness of AI-generated code for security purposes. We use ACCA to assess four state-of-the-art models trained to generate security-oriented assembly code. Our experiments show that our method outperforms the baseline solutions and assesses the correctness of the AI-generated code similar to the human-based evaluation.
  arXiv  Detail & Related papers  (2023-10-28T22:28:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.