Emergence WebVoyager: Toward Consistent and Transparent Evaluation of (Web) Agents in The Wild

• URL: http://arxiv.org/abs/2603.29020v1
• Date: Mon, 30 Mar 2026 21:27:28 GMT
• Title: Emergence WebVoyager: Toward Consistent and Transparent Evaluation of (Web) Agents in The Wild
• Abstract summary: This study identifies persistent shortcomings in existing AI agent evaluation practices.<n>We introduce Emergence WebVoyager, an enhanced version of the WebVoyager benchmark that standardizes evaluation methodology.<n>Applying this framework to evaluate OpenAI Operator reveals substantial performance variation across domains and task types.
• Score: 0.4077787659104315
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reliable evaluation of AI agents operating in complex, real-world environments requires methodologies that are robust, transparent, and contextually aligned with the tasks agents are intended to perform. This study identifies persistent shortcomings in existing AI agent evaluation practices that are particularly acute in web agent evaluation, as exemplified by our audit of WebVoyager, including task-framing ambiguity and operational variability that hinder meaningful and reproducible performance comparisons. To address these challenges, we introduce Emergence WebVoyager, an enhanced version of the WebVoyager benchmark that standardizes evaluation methodology through clear guidelines for task instantiation, failure handling, annotation, and reporting. Emergence WebVoyager achieves an inter-annotator agreement of 95.9\%, indicating improved clarity and reliability in both task formulation and evaluation. Applying this framework to evaluate OpenAI Operator reveals substantial performance variation across domains and task types, with an overall success rate of 68.6\%, substantially lower than the 87\% previously reported by OpenAI, demonstrating the utility of our approach for more rigorous and comparable web agent evaluation.

Related papers

• WebRetriever: A Large-Scale Comprehensive Benchmark for Efficient Web Agent Evaluation [26.889929086766003]
  Web agents increasingly demonstrate capabilities in automated task execution.<n>Existing benchmarks suffer from insufficient scale and limited domain diversity.<n>We introduce WebRetriever, a large-scale benchmark encompassing 800 websites and 1,550 tasks.
  arXiv  Detail & Related papers  (2026-07-07T10:27:31Z)
• Automated Benchmark Auditing for AI Agents and Large Language Models [46.03841647776303]
  Auto Benchmark Audit (ABA) is an agentic framework that systematically audits individual benchmark tasks.<n>We run ABA on a collection of frontier LLM benchmarks and previous NeurIPS publications, totaling 168 benchmarks across nine domains.<n>ABA identifies critical issues including ambiguous task design, execution environment conflicts, and incorrect ground truths in over 25.7% of the evaluated tasks.
  arXiv  Detail & Related papers  (2026-05-25T17:44:21Z)
• Talk, Evaluate, Diagnose: User-aware Agent Evaluation with Automated Error Analysis [3.3237915628874632]
  We argue that effective agent evaluation goes beyond correctness alone, incorporating conversation quality, efficiency and systematic diagnosis of agent errors.<n>We propose new metrics that capture both turn efficiency and intermediate progress of the agent.<n>We show that our TED framework reveals new insights regarding agent performance across models and user expertise levels.
  arXiv  Detail & Related papers  (2026-03-16T16:14:28Z)
• TRACE: Trajectory-Aware Comprehensive Evaluation for Deep Research Agents [51.30998248590416]
  Trajectory-Aware Comprehensive Evaluation (TRACE) is a framework that holistically assesses the entire problem-solving trajectory.<n>Our contributions include the TRACE framework, its novel metrics, and the accompanying DeepResearch-Bench with controllable complexity.
  arXiv  Detail & Related papers  (2026-02-05T13:28:57Z)
• Agentic Reward Modeling: Verifying GUI Agent via Online Proactive Interaction [7.731207237810125]
  VAGEN is a framework that employs a verifier agent equipped with interaction tools to autonomously plan verification strategies.<n>We show that VAGEN significantly improves evaluation accuracy compared to LLM-as-a-Judge baselines.
  arXiv  Detail & Related papers  (2026-01-31T07:36:54Z)
• Towards Outcome-Oriented, Task-Agnostic Evaluation of AI Agents [1.0305173936249623]
  This white paper proposes a novel framework of eleven outcome-based, task-agnostic performance metrics for AI agents.<n>We introduce metrics such as Goal Completion Rate (GCR), Autonomy Index (AIx), Multi-Step Task Resilience (MTR), and Business Impact Efficiency (BIE)<n>Our results reveal significant performance trade-offs between different agent designs, highlighting the Hybrid Agent as the most consistently high-performing model.
  arXiv  Detail & Related papers  (2025-11-11T13:40:46Z)
• WebDevJudge: Evaluating (M)LLMs as Critiques for Web Development Quality [62.43165871914528]
  We introduce WebDevJudge, a systematic benchmark for assessing LLM-as-a-judge performance in web development.<n>WebDevJudge comprises human preference labels over paired web implementations, annotated with structured and query-grounded rubrics.<n>In-depth analysis indicates this gap stems from fundamental model limitations, including failures in recognizing functional equivalence, verifying task feasibility, and mitigating bias.
  arXiv  Detail & Related papers  (2025-10-21T12:16:04Z)
• MAVUL: Multi-Agent Vulnerability Detection via Contextual Reasoning and Interactive Refinement [9.377934769326416]
  MAVUL is a novel multi-agent vulnerability detection system that integrates contextual reasoning and interactive refinement.<n>We show MAVUL significantly outperforms existing multi-agent systems with over 62% higher pairwise accuracy and single-agent systems with over 600% higher average performance.
  arXiv  Detail & Related papers  (2025-09-30T22:21:43Z)
• JudgeAgent: Knowledge-wise and Dynamic LLM Evaluation with Agent-as-Interviewer [19.09571232466437]
  We propose Agent-as-Interviewer, a dynamic evaluation paradigm for large language models (LLMs)<n>Unlike current benchmarking or dynamic interaction paradigms, Agent-as-Interviewer utilizes agents to invoke knowledge tools for wider and deeper knowledge in the dynamic multi-turn question generation.<n>We develop JudgeAgent, a knowledge-wise dynamic evaluation framework that employs knowledge-driven synthesis as the agent's tool and uses difficulty scoring as strategy guidance.
  arXiv  Detail & Related papers  (2025-09-02T08:52:16Z)
• Advancing Embodied Agent Security: From Safety Benchmarks to Input Moderation [52.83870601473094]
  Embodied agents exhibit immense potential across a multitude of domains.<n>Existing research predominantly concentrates on the security of general large language models.<n>This paper introduces a novel input moderation framework, meticulously designed to safeguard embodied agents.
  arXiv  Detail & Related papers  (2025-04-22T08:34:35Z)
• An Illusion of Progress? Assessing the Current State of Web Agents [61.742657650092845]
  We conduct a comprehensive and rigorous assessment of the current state of web agents.<n>Results depict a very different picture of the competency of current agents, suggesting over-optimism in previously reported results.<n>We introduce Online-Mind2Web, an online evaluation benchmark consisting of 300 diverse and realistic tasks spanning 136 websites.
  arXiv  Detail & Related papers  (2025-04-02T05:51:29Z)
• Interactive Agents to Overcome Ambiguity in Software Engineering [61.40183840499932]
  AI agents are increasingly being deployed to automate tasks, often based on ambiguous and underspecified user instructions.<n>Making unwarranted assumptions and failing to ask clarifying questions can lead to suboptimal outcomes.<n>We study the ability of LLM agents to handle ambiguous instructions in interactive code generation settings by evaluating proprietary and open-weight models on their performance.
  arXiv  Detail & Related papers  (2025-02-18T17:12:26Z)
• The BrowserGym Ecosystem for Web Agent Research [151.90034093362343]
  BrowserGym ecosystem addresses the growing need for efficient evaluation and benchmarking of web agents.<n>We propose an extended BrowserGym-based ecosystem for web agent research, which unifies existing benchmarks from the literature.<n>We conduct the first large-scale, multi-benchmark web agent experiment and compare the performance of 6 state-of-the-art LLMs across 6 popular web agent benchmarks.
  arXiv  Detail & Related papers  (2024-12-06T23:43:59Z)
• Autonomous Evaluation and Refinement of Digital Agents [57.12281122337407]
  We show that domain-general automatic evaluators can significantly improve the performance of agents for web navigation and device control. We validate the performance of these models in several popular benchmarks for digital agents, finding between 74.4 and 92.9% agreement with oracle evaluation metrics.
  arXiv  Detail & Related papers  (2024-04-09T17:25:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.