Odysseys: Benchmarking Web Agents on Realistic Long Horizon Tasks
- URL: http://arxiv.org/abs/2604.24964v1
- Date: Mon, 27 Apr 2026 20:05:41 GMT
- Title: Odysseys: Benchmarking Web Agents on Realistic Long Horizon Tasks
- Abstract summary: We introduce Odysseys: a benchmark of 200 long-horizon web tasks derived from real world browsing sessions evaluated on the live Internet.<n>We find that binary pass/fail evaluation is inadequate for long-horizon settings and introduce a rubric-based evaluation, annotating each Odysseys task with an average of 6.1 graded rubrics.<n>We find that the strongest models achieve a success rate of 44.5%, which leaves substantial room for future improvements.
- Score: 67.44219836008348
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Existing web agent benchmarks have largely converged on short, single-site tasks that frontier models are approaching saturation on. However, real world web use consists of long-horizon, multi-site workflows. Common web navigation tasks, such as comparing products across different domains, planning trips across multiple services, or summarizing information from multiple search queries, require sustained context and cross-site reasoning over potentially hours of browsing. To capture and evaluate such behaviors, we introduce Odysseys: a benchmark of 200 long-horizon web tasks derived from real world browsing sessions evaluated on the live Internet. We find that binary pass/fail evaluation is inadequate for long-horizon settings and introduce a rubric-based evaluation, annotating each Odysseys task with an average of 6.1 graded rubrics. We demonstrate that this yields higher agreement with humans and provides a more fine-grained signal than commonly used trajectory-level LLM-as-a-judge evaluation metrics. We tested several leading frontier models and find that the strongest models achieve a success rate of 44.5%, which leaves substantial room for future improvements. Beyond task success, we argue that efficiency is a first-class concern for long-horizon agents. We introduce a Trajectory Efficiency metric (rubric score per step) and find that even frontier agents achieve only 1.15%, marking an evident need for agents that can succeed efficiently and not simply eventually. Odysseys isolates the critical evaluation of long-horizon proficiency in open-web environments, providing a realistic benchmark to measure progress towards computer-use agents that can potentially productively operate for hours. We release our tasks, evaluation scripts, and other results at https://odysseys-website.pages.dev
Related papers
- Running the Gauntlet: Re-evaluating the Capabilities of Agents Beyond Familiar Environments [67.2234316079859]
GauntletBench is a web-based benchmark for evaluating agent generalisation in challenging scenarios.<n>It focuses on three underexplored capabilities (temporal perception, graphical understanding, and 3D reasoning) across five less-covered professional applications.<n>Our empirical results reveal that frontier agentic systems remain far from achieving human-level performance.
arXiv Detail & Related papers (2026-06-12T12:32:24Z) - SWE-Marathon: Can Agents Autonomously Complete Ultra-Long-Horizon Software Work? [11.093716663660203]
SWE-Marathon is a benchmark of 20 long-horizon tasks spanning software engineering and adjacent technical domains.<n>Each task consists of a unique executable environment, a human-written reference solution, and a multi-layer verification suite.<n>Logged agent attempts average 27.2M total tokens, making SWE-Marathon substantially longer-horizon than existing SWE and command-line agent benchmarks.
arXiv Detail & Related papers (2026-06-05T00:39:44Z) - GTA: Generating Long-Horizon Tasks for Web Agents at Scale [82.43869456830664]
We introduce a scalable framework, GTA, that integrates crawling, retrieval-based seeding, in-context generation, and automated quality control.<n>We instantiate the pipeline on over 50 websites covering e-commerce, government, forums, and news, with multilingual and multi-hop coverage.<n>Our contributions are three-fold: (i) formalizing multi-hop web-agent task generation, (ii) proposing an efficient and validated pipeline for automatic data creation, and (iii) releasing a dynamic benchmark with reproducible evaluation.
arXiv Detail & Related papers (2026-05-28T01:05:50Z) - Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces [126.23612941699565]
Terminal-Bench 2.0 is a benchmark composed of 89 tasks in computer terminal environments inspired by problems from real world.<n>We show that frontier models and agents score less than 65% on the benchmark.<n>We publish the dataset and evaluation harness to assist developers and researchers in future work at https://www.tbench.ai/.
arXiv Detail & Related papers (2026-01-17T01:29:30Z) - UltraHorizon: Benchmarking Agent Capabilities in Ultra Long-Horizon Scenarios [63.67884284105684]
We introduce textbfUltraHorizon, a novel benchmark that measures the foundational capabilities essential for complex real-world challenges.<n>Agents are designed in long-horizon discovery tasks where they must iteratively uncover hidden rules.<n>Our experiments reveal that LLM-agents consistently underperform in these settings, whereas human participants achieve higher scores.
arXiv Detail & Related papers (2025-09-26T02:04:00Z) - OdysseyBench: Evaluating LLM Agents on Long-Horizon Complex Office Application Workflows [10.318744035680398]
Large language models (LLMs) are increasingly deployed in real-world applications requiring complex, long-horizon reasoning.<n>OdysseyBench is a comprehensive benchmark for evaluating LLM agents on long-horizon across diverse office applications.<n>To enable scalable benchmark creation, we propose HomerAgents, a multi-agent framework that automates the generation of long-horizon workflow benchmarks.
arXiv Detail & Related papers (2025-08-12T17:53:03Z) - WideSearch: Benchmarking Agentic Broad Info-Seeking [26.110175228335894]
We introduce WideSearch, a new benchmark engineered to evaluate agent reliability on large-scale collection tasks.<n>The benchmark features 200 manually curated questions from over 15 diverse domains, grounded in real user queries.<n>We benchmark over 10 state-of-the-art agentic search systems, including single-agent, multi-agent frameworks, and end-to-end commercial systems.
arXiv Detail & Related papers (2025-08-11T14:03:09Z) - WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks? [83.19032025950986]
We study the use of large language model-based agents for interacting with software via web browsers.
WorkArena is a benchmark of 33 tasks based on the widely-used ServiceNow platform.
BrowserGym is an environment for the design and evaluation of such agents.
arXiv Detail & Related papers (2024-03-12T14:58:45Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.