FM-Bench: A Benchmark for Long-Horizon Management with Competing Agents

• URL: http://arxiv.org/abs/2608.18423v2
• Date: Thu, 20 Aug 2026 20:49:58 GMT
• Title: FM-Bench: A Benchmark for Long-Horizon Management with Competing Agents
• Abstract summary: FM-Bench (Football Management Benchmark) measures bounded tasks.<n>An agent runs a football club for 20 in-game years through 26 tools and roughly 340 to 400 decision stops.<n>It drafts a squad on the same budget as every rival, trades players, negotiates contracts, invests in facilities and youth, sets lineups, and answers to a board that can fire it.
• Score: 16.48804532565337
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Language model agents now execute bounded tasks reliably. Whether they can sustain effective decision-making over long horizons, where actions have cumulative consequences and the environment responds to their choices, remains largely unmeasured. FM-Bench (Football Management Benchmark) measures this. An LLM agent runs a football club for 20 in-game years through 26 tools and roughly 340 to 400 decision stops. It drafts a squad on the same budget as every rival, trades players, negotiates contracts, invests in facilities and youth, sets lineups, and answers to a board that can fire it, while a deterministic engine accumulates every year into one final score with no LLM judge or human rater. The solo track plays each of 15 frontier models against a frozen scripted world, and the Arena places the same models plus a scripted anchor in one shared 20-year world; to our knowledge, the first head-to-head evaluation at this scale. We measure six behavioral capabilities behind the score. Across three seeds, all 15 models complete every horizon while the blind scripted baselines die out in most of theirs, and claude-fable-5 tops the solo board on mean score and the Arena, where the title nonetheless rotates among ten models. Neither scale, price, nor vendor predicts the order; the order settles only late in the horizon, and the best first-play human lands only at the bottom of the model board. What separates the models is managerial behavior rather than computation. Higher-scoring models reduce slow-payoff investment near the end, keep cash invested rather than idle, and open renewals well before the deadline, while token spend predicts nothing. No model learns the market's hidden prices from hundreds of rejected bids, and self-managed memory fails in two opposite modes: an archive that only grows or a plan rewritten every season. Code is available at https://github.com/Analogy-AI/fm-bench.

Related papers

• FIFA World Cup 2026 as a Contamination-Free Benchmark for LLM Forecasting Agents: Four Models, a Bookmaker, and 104 Matches [2.8722431362974077]
  We introduce WC2026-Agents, a benchmark and dataset for evaluating large language models (LLMs)<n>For every one of the 104 matches of the 2026 FIFA World Cup, four frontier models ran an identical search-act-reflect loop.<n>We pair the four agents with a fifth competitor drawn from the same information environment -- the pre-match betting market.
  arXiv  Detail & Related papers  (2026-07-20T10:00:11Z)
• SportD: Can VLMs Physically Strategize? [3.6356011518439684]
  SportD is a benchmark comprising 478 on-ball decisions from the 2022 FIFA World Cup.<n>Each model choice is evaluated against a possession-value model that estimates the action that most increases the attacking team's probability of scoring.<n>Across three frontier VLMs, the best selects the highest-valued action on 31.4% of events, compared with 38.9% for the professional players.
  arXiv  Detail & Related papers  (2026-07-16T06:30:26Z)
• When Agents Lie: Premeditation, Persistence, and Exploitation in Repeated Games [69.060029640261]
  We evaluate three frontier models across six games in homogeneous and heterogeneous groups over 10 rounds.<n>Different models interpret announcements incompatibly, some as binding commitments and others as cheap talk.
  arXiv  Detail & Related papers  (2026-07-06T14:17:59Z)
• AgenticSTS: A Bounded-Memory Testbed for Long-Horizon LLM Agents [34.450981549319266]
  Memory for a long-horizon LLM agent is a contract about what each future decision is allowed to see.<n>We instrument an alternative bounded contract: every decision is made from a fresh user message assembled by typed retrieval.<n>We instantiate the contract in Slay the Spire 2, a closed-rule deck-building game.
  arXiv  Detail & Related papers  (2026-07-02T14:44:32Z)
• MINDGAMES: A Live Arena for Evaluating Social and Strategic Reasoning in Multi-Agent LLMs [54.81359054218573]
  We introduce Mindgames, a multi-game arena and evaluation platform for large language models (LLMs)<n>Mindgames provides a unified interaction interface, TrueSkill-based rating, and full trajectory logging across four game environments.<n>We analyze 29,571 multi-agent games with turn-level observations, actions, and rewards, together with MG-Ref, a deterministic offline tournament protocol.
  arXiv  Detail & Related papers  (2026-05-28T07:33:47Z)
• KellyBench: A Benchmark for Long-Horizon Sequential Decision Making [0.6180808157777046]
  We introduce KellyBench, an environment for evaluating sequential decision-making in sports betting markets.<n>Agents are placed in a sequential simulation of the 2023-24 English Premier League season.<n>We find that all frontier models evaluated lose money on average over the course of the season for five seeds.
  arXiv  Detail & Related papers  (2026-04-30T13:47:22Z)
• LUDOBENCH: Evaluating LLM Behavioural Decision-Making Through Spot-Based Board Game Scenarios in Ludo [1.5718921092089344]
  LudoBench is a benchmark for evaluating strategic reasoning in Ludo, a multi-agent board game.<n>LudoBench comprises 480 handcrafted spot scenarios across 12 behaviorally distinct decision categories.<n>All models agree with the game-theory baseline only 40-46% of the time.
  arXiv  Detail & Related papers  (2026-04-07T10:34:13Z)
• LM Fight Arena: Benchmarking Large Multimodal Models via Game Competition [104.81487689011341]
  We introduce LM Fight Arena, a novel framework that evaluates large multimodal models in Mortal Kombat II.<n>Unlike static evaluations, LM Fight Arena provides a fully automated, reproducible, and objective assessment of an LMM's strategic reasoning capabilities.
  arXiv  Detail & Related papers  (2025-10-10T02:19:21Z)
• Provably Efficient Generalized Lagrangian Policy Optimization for Safe Multi-Agent Reinforcement Learning [105.7510838453122]
  We examine online safe multi-agent reinforcement learning using constrained Markov games. We develop an upper confidence reinforcement learning algorithm to solve this Lagrangian problem. Our algorithm updates the minimax decision primal variables via online mirror descent and the dual variable via projected gradient step.
  arXiv  Detail & Related papers  (2023-05-31T22:09:24Z)
• ApproxED: Approximate exploitability descent via learned best responses [61.17702187957206]
  We study the problem of finding an approximate Nash equilibrium of games with continuous action sets. We propose two new methods that minimize an approximation of exploitability with respect to the strategy profile.
  arXiv  Detail & Related papers  (2023-01-20T23:55:30Z)
• Mastering the Game of Stratego with Model-Free Multiagent Reinforcement Learning [86.37438204416435]
  Stratego is one of the few iconic board games that Artificial Intelligence (AI) has not yet mastered. Decisions in Stratego are made over a large number of discrete actions with no obvious link between action and outcome. DeepNash beats existing state-of-the-art AI methods in Stratego and achieved a yearly (2022) and all-time top-3 rank on the Gravon games platform.
  arXiv  Detail & Related papers  (2022-06-30T15:53:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.