WebGames: Challenging General-Purpose Web-Browsing AI Agents

• URL: http://arxiv.org/abs/2502.18356v1
• Date: Tue, 25 Feb 2025 16:45:08 GMT
• Title: WebGames: Challenging General-Purpose Web-Browsing AI Agents
• Authors: George Thomas, Alex J. Chan, Jikun Kang, Wenqi Wu, Filippos Christianos, Fraser Greenlee, Andy Toulis, Marvin Purtorab,
• Abstract summary: WebGames is a comprehensive benchmark suite designed to evaluate general-purpose web-browsing AI agents.<n>We evaluate leading vision-language models including GPT-4o, Claude Computer-Use, Gemini-1.5-Pro, and Qwen2-VL against human performance.<n>Results reveal a substantial capability gap, with the best AI system achieving only 43.1% success rate compared to human performance of 95.7%.
• Score: 11.320069795732058
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce WebGames, a comprehensive benchmark suite designed to evaluate general-purpose web-browsing AI agents through a collection of 50+ interactive challenges. These challenges are specifically crafted to be straightforward for humans while systematically testing the limitations of current AI systems across fundamental browser interactions, advanced input processing, cognitive tasks, workflow automation, and interactive entertainment. Our framework eliminates external dependencies through a hermetic testing environment, ensuring reproducible evaluation with verifiable ground-truth solutions. We evaluate leading vision-language models including GPT-4o, Claude Computer-Use, Gemini-1.5-Pro, and Qwen2-VL against human performance. Results reveal a substantial capability gap, with the best AI system achieving only 43.1% success rate compared to human performance of 95.7%, highlighting fundamental limitations in current AI systems' ability to handle common web interaction patterns that humans find intuitive. The benchmark is publicly available at webgames.convergence.ai, offering a lightweight, client-side implementation that facilitates rapid evaluation cycles. Through its modular architecture and standardized challenge specifications, WebGames provides a robust foundation for measuring progress in development of more capable web-browsing agents.

Related papers

• REAL: Benchmarking Autonomous Agents on Deterministic Simulations of Real Websites [9.58858258192147]
  We introduce REAL, a benchmark and framework for multi-turn agent evaluations on deterministic simulations of real-world websites. We also release a benchmark consisting of 112 practical tasks that mirror everyday complex user interactions. Our framework supports easy integration of new tasks, reproducible evaluation, and scalable post-training data generation.
  arXiv  Detail & Related papers  (2025-04-15T18:22:55Z)
• BEARCUBS: A benchmark for computer-using web agents [33.1173997263462]
  BEARCUBS is a benchmark of 111 information-seeking questions designed to evaluate a web agent's ability to search, browse, and identify factual information from the web. solving BEARCUBS requires accessing live web content rather than synthetic or simulated pages, which captures the unpredictability of real-world web interactions. A human study confirms that BEARCUBS questions are solvable but non-trivial (84.7% human accuracy), revealing search inefficiencies and domain knowledge gaps as common failure points.
  arXiv  Detail & Related papers  (2025-03-10T23:50:30Z)
• The BrowserGym Ecosystem for Web Agent Research [151.90034093362343]
  BrowserGym ecosystem addresses the growing need for efficient evaluation and benchmarking of web agents.<n>We conduct the first large-scale, multi-benchmark web agent experiment.<n>Results highlight a large discrepancy between OpenAI and Anthropic's latests models.
  arXiv  Detail & Related papers  (2024-12-06T23:43:59Z)
• Visual Agents as Fast and Slow Thinkers [88.1404921693082]
  We introduce FaST, which incorporates the Fast and Slow Thinking mechanism into visual agents. FaST employs a switch adapter to dynamically select between System 1/2 modes. It tackles uncertain and unseen objects by adjusting model confidence and integrating new contextual data.
  arXiv  Detail & Related papers  (2024-08-16T17:44:02Z)
• Agent Q: Advanced Reasoning and Learning for Autonomous AI Agents [44.34340798542]
  Large Language Models (LLMs) have shown remarkable capabilities in natural language tasks requiring complex reasoning. Traditional supervised pre-training on static datasets falls short in enabling autonomous agent capabilities. We propose a framework that combines guided Monte Carlo Tree Search (MCTS) search with a self-critique mechanism and iterative fine-tuning on agent interactions.
  arXiv  Detail & Related papers  (2024-08-13T20:52:13Z)
• WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models [65.18602126334716]
  Existing web agents typically only handle one input modality and are evaluated only in simplified web simulators or static web snapshots. We introduce WebVoyager, an innovative Large Multimodal Model (LMM) powered web agent that can complete user instructions end-to-end by interacting with real-world websites. We show that WebVoyager achieves a 59.1% task success rate on our benchmark, significantly surpassing the performance of both GPT-4 (All Tools) and the WebVoyager (text-only) setups.
  arXiv  Detail & Related papers  (2024-01-25T03:33:18Z)
• WebArena: A Realistic Web Environment for Building Autonomous Agents [92.3291458543633]
  We build an environment for language-guided agents that is highly realistic and reproducible. We focus on agents that perform tasks on the web, and create an environment with fully functional websites from four common domains. We release a set of benchmark tasks focusing on evaluating the functional correctness of task completions.
  arXiv  Detail & Related papers  (2023-07-25T22:59:32Z)
• ProcTHOR: Large-Scale Embodied AI Using Procedural Generation [55.485985317538194]
  ProcTHOR is a framework for procedural generation of Embodied AI environments. We demonstrate state-of-the-art results across 6 embodied AI benchmarks for navigation, rearrangement, and arm manipulation.
  arXiv  Detail & Related papers  (2022-06-14T17:09:35Z)
• Watch-And-Help: A Challenge for Social Perception and Human-AI Collaboration [116.28433607265573]
  We introduce Watch-And-Help (WAH), a challenge for testing social intelligence in AI agents. In WAH, an AI agent needs to help a human-like agent perform a complex household task efficiently. We build VirtualHome-Social, a multi-agent household environment, and provide a benchmark including both planning and learning based baselines.
  arXiv  Detail & Related papers  (2020-10-19T21:48:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.