CEO-Bench: Can Agents Play the Long Game?

• URL: http://arxiv.org/abs/2606.18543v1
• Date: Tue, 16 Jun 2026 23:37:52 GMT
• Title: CEO-Bench: Can Agents Play the Long Game?
• Abstract summary: Language model agents are becoming proficient executors at isolated, short-horizon tasks.<n>Yet real-world challenges require a combination of sophisticated skills that remain largely untested in agents.<n>We introduce CEO-Bench, which evaluates these capabilities together by simulating a representative real-world task.
• Score: 31.540356544517593
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Language model agents are becoming proficient executors at isolated, short-horizon tasks such as software engineering and customer service. Yet real-world challenges require a combination of sophisticated skills that remain largely untested in agents: (1) navigating long horizons amid uncertainty; (2) acquiring information in noisy environments; (3) adapting to a changing world; (4) orchestrating multiple moving parts toward a coherent goal. We introduce CEO-Bench, which evaluates these capabilities together by simulating a representative real-world task: operating a startup for 500 days. An agent manages pricing, marketing, budgeting, and many other aspects of a fictional company through a programmable Python interface, operating in the same environment and facing the same challenges as a human CEO. Success demands analyzing noisy, interconnected business databases, translating signals into sound strategy, and coordinating many decisions with programming. The strongest agents write sophisticated code that simulates customer cohorts to forecast future cash and mines negotiation history to uncover hidden customer preferences. Even so, most state-of-the-art models struggle in this environment. Only Claude Opus 4.8 and GPT-5.5 finish above the $1M starting balance, and neither consistently turns a profit. CEO-Bench takes a first step toward measuring the intelligence required to drive sustained, adaptive progress over time.

Related papers

• $τ^τ$-Bench: An Environment for End-To-End, Realistic Agent Construction [20.535350290519883]
  We introduce $$-bench, a benchmark that makes agent construction the task.<n>A developer agent is given the records a business actually keeps, a client who holds requirements, a production API that operations must run through, and limits on serving cost and models.<n>Across 53 tasks spanning four domains, the strongest configuration, Claude Opus 5 under Claude Code, passes just 23.9% of evaluation simulations.
  arXiv  Detail & Related papers  (2026-09-04T01:23:47Z)
• E-Commerce Bench: Evaluating LLM Agents on Long-Horizon Autonomous Business Operation [54.50499240166176]
  We introduce E-Commerce Bench, the first open-source benchmark that integrates multi-round counterpart negotiation and dynamic events into a year-long business operation.<n>We evaluate 18 frontier models across seven dimensions, including year-end assets, and find that no single model dominates.
  arXiv  Detail & Related papers  (2026-08-31T13:03:57Z)
• SWE-INTERACT: Reimagining SWE Benchmarks as User-Driven Long-Horizon Coding Sessions [3.67653051806469]
  SWE-Interact places agents in a realistic developer workflow.<n>It starts with vague or incomplete instructions, progressively reveals requirements.<n>It provides targeted feedback, revisions, and new constraints until the full task goal has been handed off.
  arXiv  Detail & Related papers  (2026-06-29T17:17:45Z)
• Benchmarking Open-Ended Multi-Agent Coordination in Language Agents [48.1363632826625]
  Alem embeds procedurally generated coordination tasks, soft specialisation, communication, and controllable coordination difficulty into a long-horizon survival world.<n>We introduce $alem$, a JAX-based benchmark for open-ended multi-agent coordination built on Craftax-like dynamics.
  arXiv  Detail & Related papers  (2026-06-06T21:13:43Z)
• $\texttt{YC-Bench}$: Benchmarking AI Agents for Long-Term Planning and Consistent Execution [38.30265058791818]
  We introduce $texttYC-Bench$, a benchmark that evaluates capabilities by tasking an agent with running a simulated startup over a one-year horizon spanning hundreds of turns.<n>We evaluate 12 models, both proprietary and open source, across 3 seeds each.<n>Only three models consistently surpass the starting capital of $200K, with Claude Opus 4.6 achieving the highest average final funds at $1.27 M, followed by GLM-5 at $1.21 M at 11$times$ lower inference cost.
  arXiv  Detail & Related papers  (2026-04-01T17:52:19Z)
• SWE-Bench Mobile: Can Large Language Model Agents Develop Industry-Level Mobile Applications? [21.241252187534055]
  SWE-Bench Mobile is a benchmark for evaluating coding agents on realistic software engineering tasks derived from a production iOS.<n>Unlike existing benchmarks that focus on isolated problems or bug fixes, SWE-Bench Mobile captures the full complexity of industrial development.
  arXiv  Detail & Related papers  (2026-02-10T08:51:11Z)
• Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation [87.47155146067962]
  We provide a standardized evaluation harness that orchestrates parallel evaluations across hundreds of tasks.<n>We conduct three-dimensional analysis spanning models, scaffolds, and benchmarks.<n>Our analysis reveals surprising insights, such as higher reasoning effort reducing accuracy in the majority of runs.
  arXiv  Detail & Related papers  (2025-10-13T22:22:28Z)
• TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks [55.03911355902567]
  We introduce TheAgentCompany, a benchmark for evaluating AI agents that interact with the world in similar ways to those of a digital worker.<n>We find that the most competitive agent can complete 30% of tasks autonomously.<n>This paints a nuanced picture on task automation with simulating LM agents in a setting a real workplace.
  arXiv  Detail & Related papers  (2024-12-18T18:55:40Z)
• Richelieu: Self-Evolving LLM-Based Agents for AI Diplomacy [24.521882655442187]
  Diplomacy involves a staggering magnitude of decision spaces, especially considering the negotiation stage required. Previous AI agents have demonstrated their ability to handle multi-step games and large action spaces in multi-agent tasks. We aim to explore AI's potential to create a human-like agent capable of executing comprehensive multi-agent missions.
  arXiv  Detail & Related papers  (2024-07-09T12:37:54Z)
• AgentGym: Evolving Large Language Model-based Agents across Diverse Environments [116.97648507802926]
  Large language models (LLMs) are considered a promising foundation to build such agents. We take the first step towards building generally-capable LLM-based agents with self-evolution ability. We propose AgentGym, a new framework featuring a variety of environments and tasks for broad, real-time, uni-format, and concurrent agent exploration.
  arXiv  Detail & Related papers  (2024-06-06T15:15:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.