Agents' Last Exam

• URL: http://arxiv.org/abs/2606.05405v2
• Date: Thu, 11 Jun 2026 10:09:39 GMT
• Title: Agents' Last Exam
• Abstract summary: Agents' Last Exam (ALE) is a benchmark designed to evaluate AI agents on long horizon, economically valuable, real world tasks with verifiable outcomes.<n>Developed in collaboration with 250+ industry experts, ALE covers non-physical industries defined with reference to O*NET / SOC 2018.<n>Current results show that the hardest tier remains far from saturated: across mainstream harness and backbone configurations, the average full pass rate is below 1%.
• Score: 275.79071391137074
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent AI systems have achieved strong results on a wide range of benchmarks, yet these gains have not translated into economically meaningful deployment across many professional domains. We argue that this gap is largely an evaluation problem: widely used benchmarks lack sustained performance measurement on real and economically valuable workflows. This paper introduces Agents' Last Exam (ALE), a benchmark designed to evaluate AI agents on long horizon, economically valuable, real world tasks with verifiable outcomes. Developed in collaboration with 250+ industry experts, ALE covers non-physical industries defined with reference to O*NET / SOC 2018 (the U.S. federal occupational taxonomy). It is organized around a task taxonomy with 55 sub fields grouped into 13 industry clusters covering 1K+ tasks. Current results show that the hardest tier remains far from saturated: across mainstream harness and backbone configurations, the average full pass rate is below 1%. ALE is designed as a living benchmark: its task pool grows continuously as new workflows and industries are onboarded. More broadly, ALE is intended not merely as another leaderboard, but as an instrument for closing the gap between benchmark success and GDP relevant impact.

Related papers

• StartupBench: Benchmarking General-Purpose Agents on Market-Validated End-to-End Workflows [63.57092302641323]
  We introduce textbfStartupBench, an E2E agent benchmark grounded in market-validated AI startup products.<n>We study AI products with demonstrated adoption, together with their product and users, to identify real-world tasks for which AI has established practical demand.<n>Our results reveal that many market-validated agents remain beyond the reliable capabilities of current general-purpose agents.
  arXiv  Detail & Related papers  (2026-08-18T14:01:32Z)
• Welfare, Improvability, and Variance: A Principal-Agent Approach to Optimal Benchmark Item Aggregation [19.980368400366352]
  We model benchmarking as a multitask principal-agent game and show that the welfare loss from a benchmark is determined jointly by three item-level primitives.<n>We apply the theory to OLMES items using WORKBank for welfare, the EvoLM 4B suite for improvability, and the PolyPythias 410M panel for variance.
  arXiv  Detail & Related papers  (2026-05-29T07:01:38Z)
• BankerToolBench: Evaluating AI Agents in End-to-End Investment Banking Workflows [12.076413424824679]
  Existing AI benchmarks lack the fidelity to assess economically meaningful progress on professional.<n>To evaluate frontier AI agents in a high-value, labor-intensive profession, we introduce BankerToolBench (BTB)<n>BTB is an open-source benchmark of end-to-end analytical deliverables routinely performed by junior investment bankers.
  arXiv  Detail & Related papers  (2026-04-13T11:02:32Z)
• OccuBench: Evaluating AI Agents on Real-World Professional Tasks via Language Environment Simulation [57.505743202759646]
  OccuBench is a benchmark covering 100 real-world professional task scenarios across 10 industry categories and 65 specialized domains.<n>Our multi-agent synthesis pipeline automatically produces evaluation instances with guaranteed solvability, calibrated difficulty, and document-grounded diversity.
  arXiv  Detail & Related papers  (2026-04-13T00:27:32Z)
• COMPOSITE-Stem [32.17652486099204]
  COMPOSITE-STEM is a benchmark of 70 expert-written tasks in physics, biology, chemistry, and mathematics.<n>Our benchmark combines exact-match grading and criterion-based rubrics with an-as-a-jury grading protocol.<n>The top-performing model achieves 21%, demonstrating that COMPOSITE-STEM captures capabilities beyond current agent reach.
  arXiv  Detail & Related papers  (2026-04-10T19:08:50Z)
• The AI Transformation Gap Index (AITG): An Empirical Framework for Measuring AI Transformation Opportunity, Disruption Risk, and Value Creation at the Industry and Firm Level [0.0]
  This paper introduces the AI Transformation Gap Index (AITG), a composite empirical framework that measures the distance between a firm's current AI deployment and a time varying, industry constrained capability frontier.<n>I calibrate the framework for 22 industry verticals and apply it to 14 public companies using public filings.<n>A counterintuitive result emerges: the largest AI transformation gaps do not produce the highest value density, because implementation friction, CES bottlenecks, and timing lags erode the theoretical upside of wide gaps.
  arXiv  Detail & Related papers  (2026-02-27T10:54:58Z)
• AIRS-Bench: a Suite of Tasks for Frontier AI Research Science Agents [49.67355440164857]
  We introduce AIRS-Bench, a suite of 20 tasks sourced from state-of-the-art machine learning papers.<n>Airs-Bench tasks assess agentic capabilities over the full research lifecycle.<n>We open-source the AIRS-Bench task definitions and evaluation code to catalyze further development in autonomous scientific research.
  arXiv  Detail & Related papers  (2026-02-06T16:45:02Z)
• WideSearch: Benchmarking Agentic Broad Info-Seeking [26.110175228335894]
  We introduce WideSearch, a new benchmark engineered to evaluate agent reliability on large-scale collection tasks.<n>The benchmark features 200 manually curated questions from over 15 diverse domains, grounded in real user queries.<n>We benchmark over 10 state-of-the-art agentic search systems, including single-agent, multi-agent frameworks, and end-to-end commercial systems.
  arXiv  Detail & Related papers  (2025-08-11T14:03:09Z)
• FieldWorkArena: Agentic AI Benchmark for Real Field Work Tasks [52.47895046206854]
  FieldWorkArena is a benchmark for agentic AI targeting real-world field work.<n>This paper defines a new action space that agentic AI should possess for real world work environment benchmarks.
  arXiv  Detail & Related papers  (2025-05-26T08:21:46Z)
• The BrowserGym Ecosystem for Web Agent Research [151.90034093362343]
  BrowserGym ecosystem addresses the growing need for efficient evaluation and benchmarking of web agents.<n>We propose an extended BrowserGym-based ecosystem for web agent research, which unifies existing benchmarks from the literature.<n>We conduct the first large-scale, multi-benchmark web agent experiment and compare the performance of 6 state-of-the-art LLMs across 6 popular web agent benchmarks.
  arXiv  Detail & Related papers  (2024-12-06T23:43:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.