WebRetriever: A Large-Scale Comprehensive Benchmark for Efficient Web Agent Evaluation

• URL: http://arxiv.org/abs/2607.06118v1
• Date: Tue, 07 Jul 2026 10:27:31 GMT
• Title: WebRetriever: A Large-Scale Comprehensive Benchmark for Efficient Web Agent Evaluation
• Abstract summary: Web agents increasingly demonstrate capabilities in automated task execution.<n>Existing benchmarks suffer from insufficient scale and limited domain diversity.<n>We introduce WebRetriever, a large-scale benchmark encompassing 800 websites and 1,550 tasks.
• Score: 26.889929086766003
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: As web agents increasingly demonstrate capabilities in automated task execution, the development of robust evaluation frameworks for assessing their navigation and task completion performance has emerged as a critical research priority. However, existing benchmarks exhibit fundamental limitations. First, they suffer from insufficient scale and limited domain diversity, constraining comprehensive evaluation of cross-domain generalization. Second, prevailing LLM-as-Judge evaluation methodologies inadequately capture fine-grained interaction semantics, particularly regarding precise query formulation and filtering operations. Third, current benchmarks predominantly emphasize navigation success metrics while neglecting critical requirements for real-world deployment scenarios. To address these limitations, we introduce WebRetriever, a large-scale benchmark encompassing 800 websites and 1,550 tasks across diverse domains, including consumer, professional, and enterprise sectors, with comprehensive coverage of user intent patterns. We propose NavEval (Navigation Evaluation), a novel LLM-as-Judge framework that leverages rich interaction context beyond visual screenshots, achieving state-of-the-art alignment with human judgment across multiple evaluation datasets. Furthermore, we establish three complementary evaluation protocols that collectively provide holistic assessment of web agent capabilities: navigation proficiency, knowledge-assisted interaction, and end-to-end task completion with information extraction. Extensive experimental analysis reveals substantial performance disparities across evaluation protocols, demonstrating that navigation success alone is an insufficient predictor of real-world application effectiveness. WebRetriever delivers fine-grained diagnostic insights into agent capabilities and establishes a rigorous foundation for advancing web agent research and development.

Related papers

• CAP: A Scalable Benchmark for Evaluating Cross-Site Browser Agents with Complex Actions and Perception [22.620664364045286]
  Large language models are increasingly deployed as autonomous agents that interact with the web through browsers.<n>We introduce CAP, a scalable benchmark for evaluating browser agents on cross-site, human-like web tasks.<n>We construct 420 tasks across 108 real-world websites and 24 domains under careful quality control.
  arXiv  Detail & Related papers  (2026-08-09T01:08:19Z)
• Running the Gauntlet: Re-evaluating the Capabilities of Agents Beyond Familiar Environments [67.2234316079859]
  GauntletBench is a web-based benchmark for evaluating agent generalisation in challenging scenarios.<n>It focuses on three underexplored capabilities (temporal perception, graphical understanding, and 3D reasoning) across five less-covered professional applications.<n>Our empirical results reveal that frontier agentic systems remain far from achieving human-level performance.
  arXiv  Detail & Related papers  (2026-06-12T12:32:24Z)
• GTA: Generating Long-Horizon Tasks for Web Agents at Scale [82.43869456830664]
  We introduce a scalable framework, GTA, that integrates crawling, retrieval-based seeding, in-context generation, and automated quality control.<n>We instantiate the pipeline on over 50 websites covering e-commerce, government, forums, and news, with multilingual and multi-hop coverage.<n>Our contributions are three-fold: (i) formalizing multi-hop web-agent task generation, (ii) proposing an efficient and validated pipeline for automatic data creation, and (iii) releasing a dynamic benchmark with reproducible evaluation.
  arXiv  Detail & Related papers  (2026-05-28T01:05:50Z)
• MiroEval: Benchmarking Multimodal Deep Research Agents in Process and Outcome [109.15093810810214]
  MiroEval is a benchmark and evaluation framework for deep research systems.<n>The benchmark comprises 100 tasks grounded in real user needs.<n>The proposed evaluation suite assesses deep research systems along three complementary dimensions.
  arXiv  Detail & Related papers  (2026-03-30T13:16:03Z)
• Scaling Web Agent Training through Automatic Data Generation and Fine-grained Evaluation [54.945281159783896]
  We present a scalable pipeline for automatically generating high-quality training data for web agents.<n>We introduce a novel constraint-based evaluation framework that provides fine-grained assessment of progress towards task completion.
  arXiv  Detail & Related papers  (2026-02-13T02:52:18Z)
• Benchmarking Agents in Insurance Underwriting Environments [0.9728664856449597]
  Existing benchmarks overemphasize open-domains such as code, use narrow accuracy metrics, and lack authentic complexity.<n>We present UNDERWRITE, an expert-first, multi-turn insurance underwriting benchmark designed in close collaboration with domain experts.
  arXiv  Detail & Related papers  (2026-01-31T02:12:11Z)
• DeepSearchQA: Bridging the Comprehensiveness Gap for Deep Research Agents [10.197402632091551]
  DeepSearchQA is a 900-prompt benchmark for evaluating agents on difficult multi-step information-seeking tasks.<n>This dataset is designed to evaluate an agent's ability to execute complex search plans to generate exhaustive answer lists.
  arXiv  Detail & Related papers  (2026-01-28T19:20:47Z)
• WebDevJudge: Evaluating (M)LLMs as Critiques for Web Development Quality [62.43165871914528]
  We introduce WebDevJudge, a systematic benchmark for assessing LLM-as-a-judge performance in web development.<n>WebDevJudge comprises human preference labels over paired web implementations, annotated with structured and query-grounded rubrics.<n>In-depth analysis indicates this gap stems from fundamental model limitations, including failures in recognizing functional equivalence, verifying task feasibility, and mitigating bias.
  arXiv  Detail & Related papers  (2025-10-21T12:16:04Z)
• REAL: Benchmarking Autonomous Agents on Deterministic Simulations of Real Websites [9.58858258192147]
  We introduce REAL, a benchmark and framework for multi-turn agent evaluations on deterministic simulations of real-world websites.<n>We also release a benchmark consisting of 112 practical tasks that mirror everyday complex user interactions.<n>Our framework supports easy integration of new tasks, reproducible evaluation, and scalable post-training data generation.
  arXiv  Detail & Related papers  (2025-04-15T18:22:55Z)
• An Illusion of Progress? Assessing the Current State of Web Agents [61.742657650092845]
  We conduct a comprehensive and rigorous assessment of the current state of web agents.<n>Results depict a very different picture of the competency of current agents, suggesting over-optimism in previously reported results.<n>We introduce Online-Mind2Web, an online evaluation benchmark consisting of 300 diverse and realistic tasks spanning 136 websites.
  arXiv  Detail & Related papers  (2025-04-02T05:51:29Z)
• AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents [74.16170899755281]
  We introduce AgentBoard, a pioneering comprehensive benchmark and accompanied open-source evaluation framework tailored to analytical evaluation of LLM agents.<n>AgentBoard offers a fine-grained progress rate metric that captures incremental advancements as well as a comprehensive evaluation toolkit.<n>This not only sheds light on the capabilities and limitations of LLM agents but also propels the interpretability of their performance to the forefront.
  arXiv  Detail & Related papers  (2024-01-24T01:51:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.