CAP: A Scalable Benchmark for Evaluating Cross-Site Browser Agents with Complex Actions and Perception

• URL: http://arxiv.org/abs/2608.08392v1
• Date: Sun, 09 Aug 2026 01:08:19 GMT
• Title: CAP: A Scalable Benchmark for Evaluating Cross-Site Browser Agents with Complex Actions and Perception
• Abstract summary: Large language models are increasingly deployed as autonomous agents that interact with the web through browsers.<n>We introduce CAP, a scalable benchmark for evaluating browser agents on cross-site, human-like web tasks.<n>We construct 420 tasks across 108 real-world websites and 24 domains under careful quality control.
• Score: 22.620664364045286
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models are increasingly deployed as autonomous agents that interact with the web through browsers. While recent progress has been driven by benchmarks that evaluate end-to-end task success, these evaluations largely overlook two fundamental sources of difficulty in real web browsing: complex actions over rich user interfaces and visual perception of dynamically rendered content, especially in workflows that span multiple websites. We introduce CAP, a scalable benchmark for evaluating browser agents on cross-site, human-like web tasks that require non-trivial UI interactions and visual understanding. Specifically, we adopt a decomposition-and-recomposition pipeline that first abstracts each website into a structured site card capturing user-facing functions, complex execution operations, and perceptual requirements, and then recomposes these components into realistic cross-site workflows. Each task is therefore grounded in multiple specific operations on each website, enabling fine-grained diagnosis. Built on this framework, we construct 420 tasks across 108 real-world websites and 24 domains under careful quality control. Experiments on state-of-the-art browser agents using our verifiable agent-as-a-judge evaluation framework show low success rates and reveal that perception-heavy interactions remain a major bottleneck, exposing substantial gaps between current agents and real-world web browsing demands.

Related papers

• WebRetriever: A Large-Scale Comprehensive Benchmark for Efficient Web Agent Evaluation [26.889929086766003]
  Web agents increasingly demonstrate capabilities in automated task execution.<n>Existing benchmarks suffer from insufficient scale and limited domain diversity.<n>We introduce WebRetriever, a large-scale benchmark encompassing 800 websites and 1,550 tasks.
  arXiv  Detail & Related papers  (2026-07-07T10:27:31Z)
• Running the Gauntlet: Re-evaluating the Capabilities of Agents Beyond Familiar Environments [67.2234316079859]
  GauntletBench is a web-based benchmark for evaluating agent generalisation in challenging scenarios.<n>It focuses on three underexplored capabilities (temporal perception, graphical understanding, and 3D reasoning) across five less-covered professional applications.<n>Our empirical results reveal that frontier agentic systems remain far from achieving human-level performance.
  arXiv  Detail & Related papers  (2026-06-12T12:32:24Z)
• GTA: Generating Long-Horizon Tasks for Web Agents at Scale [82.43869456830664]
  We introduce a scalable framework, GTA, that integrates crawling, retrieval-based seeding, in-context generation, and automated quality control.<n>We instantiate the pipeline on over 50 websites covering e-commerce, government, forums, and news, with multilingual and multi-hop coverage.<n>Our contributions are three-fold: (i) formalizing multi-hop web-agent task generation, (ii) proposing an efficient and validated pipeline for automatic data creation, and (iii) releasing a dynamic benchmark with reproducible evaluation.
  arXiv  Detail & Related papers  (2026-05-28T01:05:50Z)
• WebSP-Eval: Evaluating Web Agents on Website Security and Privacy Tasks [10.738768990548428]
  We introduce WebSP-Eval, an evaluation framework for measuring web agent performance on website security and privacy tasks.<n>WebSP-Eval comprises 1) a manually crafted task dataset of 200 task instances across 28 websites; 2) a robust agentic system supporting account and initial state management across runs using a custom Google Chrome extension; and 3) an automated evaluator.<n>We evaluate a total of 8 web agent instantiations using state-of-the-art multimodal large language models, conducting a fine-grained analysis across websites, task categories, and UI elements.
  arXiv  Detail & Related papers  (2026-04-07T18:43:21Z)
• Vision2Web: A Hierarchical Benchmark for Visual Website Development with Agent Verification [27.677130026020006]
  Vision2Web is a hierarchical benchmark for visual website development spanning from static UI-to-code generation to full-stack development.<n>The benchmark comprises a total of 193 tasks across 16 categories, with 918 prototype images and 1,255 test cases.<n>We evaluate multiple visual language models instantiated under different coding-agent frameworks, revealing substantial performance gaps at all task levels.
  arXiv  Detail & Related papers  (2026-03-27T17:50:45Z)
• Nested Browser-Use Learning for Agentic Information Seeking [60.775556172513014]
  Information-seeking (IS) agents have achieved strong performance across a range of wide and deep search tasks, yet their tool use remains largely restricted to API-level snippet retrieval and URL-based page fetching.<n>We propose Nested Browser-Use Learning (NestBrowse), which introduces a minimal and complete browser-action framework that decouples interaction control from page exploration through a nested structure.
  arXiv  Detail & Related papers  (2025-12-29T17:59:14Z)
• BrowserArena: Evaluating LLM Agents on Real-World Web Navigation Tasks [51.803138848305814]
  We introduce BrowserArena, a live open-web agent evaluation platform that collects user-submitted tasks.<n>We identify three consistent failure modes: captcha resolution, pop-up banner removal, and direct navigation to URLs.<n>Our findings surface both the diversity and brittleness of current web agents.
  arXiv  Detail & Related papers  (2025-10-02T15:22:21Z)
• IWR-Bench: Can LVLMs reconstruct interactive webpage from a user interaction video? [56.33950760097989]
  IWR-Bench is a novel benchmark for evaluating the capabilities of Large Vision-Language Models (LVLMs) in interactive webpage reconstruction from video.<n>IWR-Bench comprises 113 meticulously curated tasks from 100 real-world websites, with 1,001 actions.<n>This benchmark evaluates models on two fundamental challenges: comprehensive multi-modal reasoning to infer interaction logic from video and assets, and advanced code generation to translate this logic into functional code.
  arXiv  Detail & Related papers  (2025-09-29T12:38:06Z)
• REAL: Benchmarking Autonomous Agents on Deterministic Simulations of Real Websites [9.58858258192147]
  We introduce REAL, a benchmark and framework for multi-turn agent evaluations on deterministic simulations of real-world websites.<n>We also release a benchmark consisting of 112 practical tasks that mirror everyday complex user interactions.<n>Our framework supports easy integration of new tasks, reproducible evaluation, and scalable post-training data generation.
  arXiv  Detail & Related papers  (2025-04-15T18:22:55Z)
• The BrowserGym Ecosystem for Web Agent Research [151.90034093362343]
  BrowserGym ecosystem addresses the growing need for efficient evaluation and benchmarking of web agents.<n>We propose an extended BrowserGym-based ecosystem for web agent research, which unifies existing benchmarks from the literature.<n>We conduct the first large-scale, multi-benchmark web agent experiment and compare the performance of 6 state-of-the-art LLMs across 6 popular web agent benchmarks.
  arXiv  Detail & Related papers  (2024-12-06T23:43:59Z)
• From Context to Action: Analysis of the Impact of State Representation and Context on the Generalization of Multi-Turn Web Navigation Agents [7.41862656697588]
  This study aims to analyze the various contextual elements crucial to the functioning of web navigation agents. We focus on the influence of interaction history and web page representation. Our work highlights improved agent performance across out-of-distribution scenarios.
  arXiv  Detail & Related papers  (2024-10-31T01:51:41Z)
• VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks [93.85005277463802]
  VisualWebArena is a benchmark designed to assess the performance of multimodal web agents on realistic tasks. To perform on this benchmark, agents need to accurately process image-text inputs, interpret natural language instructions, and execute actions on websites to accomplish user-defined objectives.
  arXiv  Detail & Related papers  (2024-01-24T18:35:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.