WebTestBench: Evaluating Computer-Use Agents towards End-to-End Automated Web Testing

• URL: http://arxiv.org/abs/2603.25226v1
• Date: Thu, 26 Mar 2026 09:27:29 GMT
• Title: WebTestBench: Evaluating Computer-Use Agents towards End-to-End Automated Web Testing
• Abstract summary: We introduce WebTestBench, a benchmark for evaluating end-to-end automated web testing.<n>We decompose the testing process into two cascaded sub-tasks, checklist generation and defect detection, and propose WebTester.<n>Our findings expose a substantial gap between current computer-use agent capabilities and industrial-grade deployment demands.
• Score: 57.7131457251794
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The emergence of Large Language Models (LLMs) has catalyzed a paradigm shift in programming, giving rise to "vibe coding", where users can build complete projects and even control computers using natural language instructions. This paradigm has driven automated webpage development, but it introduces a new requirement about how to automatically verify whether the web functionalities are reliably implemented. Existing works struggle to adapt, relying on static visual similarity or predefined checklists that constrain their utility in open-ended environments. Furthermore, they overlook a vital aspect of software quality, namely latent logical constraints. To address these gaps, we introduce WebTestBench, a benchmark for evaluating end-to-end automated web testing. WebTestBench encompasses comprehensive dimensions across diverse web application categories. We decompose the testing process into two cascaded sub-tasks, checklist generation and defect detection, and propose WebTester, a baseline framework for this task. Evaluating popular LLMs with WebTester reveals severe challenges, including insufficient test completeness, detection bottlenecks, and long-horizon interaction unreliability. These findings expose a substantial gap between current computer-use agent capabilities and industrial-grade deployment demands. We hope that WebTestBench provides valuable insights and guidance for advancing end-to-end automated web testing. Our dataset and code are available at https://github.com/friedrichor/WebTestBench.

Related papers

• Framework and Benchmark for Code-Driven Agentic Testing in Web Development [25.642063881090337]
  We introduce textbfCode-driven textbfAgentic textbfTesting (CAT), a paradigm in which the agent writes Playwright code to drive the browser, gathers feedback, and autonomously explores web applications to uncover bugs.<n>We instantiate CAT with CATJudge, an agentic framework that unifies Browser-Use and Computer-Use tools within a single environment and CATTest, a benchmark of 102 AI-generated web applications with carefully annotated bugs.
  arXiv  Detail & Related papers  (2026-08-31T10:07:51Z)
• LabOSBench: Benchmarking Computer Use Agents for Scientific Instrument Control [46.674982266623196]
  LabOSBench is a benchmark for multimodal GUI agents built on a suite of web-based scientific-instrument simulators.<n>We evaluate general-purpose vision-language models, specialized GUI agent models, and advanced agentic frameworks at both subtask and end-to-end levels.
  arXiv  Detail & Related papers  (2026-06-15T14:42:33Z)
• GTA: Generating Long-Horizon Tasks for Web Agents at Scale [82.43869456830664]
  We introduce a scalable framework, GTA, that integrates crawling, retrieval-based seeding, in-context generation, and automated quality control.<n>We instantiate the pipeline on over 50 websites covering e-commerce, government, forums, and news, with multilingual and multi-hop coverage.<n>Our contributions are three-fold: (i) formalizing multi-hop web-agent task generation, (ii) proposing an efficient and validated pipeline for automatic data creation, and (iii) releasing a dynamic benchmark with reproducible evaluation.
  arXiv  Detail & Related papers  (2026-05-28T01:05:50Z)
• Finetuning LLMs for Automatic Form Interaction on Web-Browser in Selenium Testing Framework [4.53273595732354]
  This paper introduces a novel method for training large language models (LLMs) to generate high-quality test cases in Selenium.<n>We curate both synthetic and human-annotated datasets for training and evaluation, covering diverse real-world forms and testing scenarios.<n>Our approach significantly outperforms strong baselines, including GPT-4o and other popular LLMs, across all evaluation metrics.
  arXiv  Detail & Related papers  (2025-11-19T06:43:21Z)
• AUTO-Explorer: Automated Data Collection for GUI Agent [58.58097564914626]
  We propose an automated data collection method with minimal annotation costs, named Auto-Explorer.<n>It incorporates a simple yet effective exploration mechanism that autonomously parses and explores GUI environments.<n>Using the data gathered, we fine-tune a multimodal large language model (MLLM) and establish a GUI element grounding testing set.
  arXiv  Detail & Related papers  (2025-11-09T15:13:45Z)
• ImpossibleBench: Measuring LLMs' Propensity of Exploiting Test Cases [58.411135609139855]
  "Shortcuts" to complete tasks pose significant risks for reliable assessment and deployment of large language models.<n>We introduce ImpossibleBench, a benchmark framework that measures LLM agents' propensity to exploit test cases.<n>As a practical framework, ImpossibleBench is not just an evaluation but a versatile tool.
  arXiv  Detail & Related papers  (2025-10-23T06:58:32Z)
• AI Agents for Web Testing: A Case Study in the Wild [20.669140680308494]
  We present WebProber, a prototype AI agent-based web testing framework.<n>Given a URL, WebProber autonomously explores the website, simulating real user interactions, identifying bugs and usability issues, and producing a human-readable report.<n>We evaluate WebProber through a case study of 120 academic personal websites, where it uncovered 29 usability issues--many of which were missed by traditional tools.
  arXiv  Detail & Related papers  (2025-09-05T15:57:16Z)
• Automated Web Application Testing: End-to-End Test Case Generation with Large Language Models and Screen Transition Graphs [0.5965410190046627]
  This paper presents an automated system for generating test cases for two key aspects of web application testing: site navigation and form filling.<n>For site navigation, the system employs screen transition graphs and LLMs to model navigation flows and generate test scenarios.<n>For form filling, it uses state graphs to handle conditional forms and automates Selenium script generation.
  arXiv  Detail & Related papers  (2025-06-03T07:08:21Z)
• TESTQUEST: A Web Gamification Tool to Improve Locators and Page Objects Quality [2.156170153103442]
  TestQUEST is a tool designed to improve test robustness by applying to locators and Page Objects.<n> locators are highly sensitive to the frequent changes in Web page structures caused by rapid software evolution.
  arXiv  Detail & Related papers  (2025-05-30T16:18:10Z)
• The BrowserGym Ecosystem for Web Agent Research [151.90034093362343]
  BrowserGym ecosystem addresses the growing need for efficient evaluation and benchmarking of web agents.<n>We propose an extended BrowserGym-based ecosystem for web agent research, which unifies existing benchmarks from the literature.<n>We conduct the first large-scale, multi-benchmark web agent experiment and compare the performance of 6 state-of-the-art LLMs across 6 popular web agent benchmarks.
  arXiv  Detail & Related papers  (2024-12-06T23:43:59Z)
• AutoPT: How Far Are We from the End2End Automated Web Penetration Testing? [54.65079443902714]
  We introduce AutoPT, an automated penetration testing agent based on the principle of PSM driven by LLMs. Our results show that AutoPT outperforms the baseline framework ReAct on the GPT-4o mini model.
  arXiv  Detail & Related papers  (2024-11-02T13:24:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.