KellyBench: A Benchmark for Long-Horizon Sequential Decision Making

• URL: http://arxiv.org/abs/2604.27865v1
• Date: Thu, 30 Apr 2026 13:47:22 GMT
• Title: KellyBench: A Benchmark for Long-Horizon Sequential Decision Making
• Abstract summary: We introduce KellyBench, an environment for evaluating sequential decision-making in sports betting markets.<n>Agents are placed in a sequential simulation of the 2023-24 English Premier League season.<n>We find that all frontier models evaluated lose money on average over the course of the season for five seeds.
• Score: 0.6180808157777046
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Language models are saturating benchmarks for procedural tasks with narrow objectives. But they are increasingly being deployed in long-horizon, non-stationary environments with open-ended goals. In this paper we introduce KellyBench, an environment for evaluating sequential decision-making in sports betting markets. Agents are placed in a sequential simulation of the 2023-24 English Premier League season and tasked with maximising their long-term bankroll growth. They are given detailed historical data, including advanced statistics, lineups, and public odds. To succeed they must build machine learning models, identify edge in public markets, and adapt as the environment changes over time. We find that all frontier models evaluated lose money on average over the course of the season for five seeds. The best performing model achieves an average return of -8%, and many models experiencing ruin across seeds. To judge strategy sophistication, we use a human expert rubric to grade each model and find their approaches to be unsophisticated compared to human baselines; Claude Opus 4.6 achieves a rubric score of 26.5%, which means there is significant room for improvement. KellyBench is available as an open-access API endpoint at https://openreward.ai/GeneralReasoning/KellyBench.

Related papers

• E-Commerce Bench: Evaluating LLM Agents on Long-Horizon Autonomous Business Operation [54.50499240166176]
  We introduce E-Commerce Bench, the first open-source benchmark that integrates multi-round counterpart negotiation and dynamic events into a year-long business operation.<n>We evaluate 18 frontier models across seven dimensions, including year-end assets, and find that no single model dominates.
  arXiv  Detail & Related papers  (2026-08-31T13:03:57Z)
• One Capability or Many? Testing the Economic Validity of Frontier AI Evaluation [0.0]
  We test whether benchmarks measure capability distinct from general test-taking.<n>Economic benchmarks add incremental predictive information to a largely date-driven general factor.<n> Leaderboards remain a sound guide to overall progress, but most of the gap between models released months apart is calendar.
  arXiv  Detail & Related papers  (2026-08-29T19:55:20Z)
• FM-Bench: A Benchmark for Long-Horizon Management with Competing Agents [16.48804532565337]
  FM-Bench (Football Management Benchmark) measures bounded tasks.<n>An agent runs a football club for 20 in-game years through 26 tools and roughly 340 to 400 decision stops.<n>It drafts a squad on the same budget as every rival, trades players, negotiates contracts, invests in facilities and youth, sets lineups, and answers to a board that can fire it.
  arXiv  Detail & Related papers  (2026-08-19T01:33:36Z)
• FIFA World Cup 2026 as a Contamination-Free Benchmark for LLM Forecasting Agents: Four Models, a Bookmaker, and 104 Matches [2.8722431362974077]
  We introduce WC2026-Agents, a benchmark and dataset for evaluating large language models (LLMs)<n>For every one of the 104 matches of the 2026 FIFA World Cup, four frontier models ran an identical search-act-reflect loop.<n>We pair the four agents with a fifth competitor drawn from the same information environment -- the pre-match betting market.
  arXiv  Detail & Related papers  (2026-07-20T10:00:11Z)
• SpatialBench: Is Your Spatial Foundation Model an All-Round Player? [92.031716744172]
  SpatialBench is a cross-paradigm, domain-diverse benchmark for spatial foundation models with deterministic sampling.<n>It comprehensively evaluates 41 models across 6 paradigms on 5 task suites under 4 different input density settings.<n>Our evaluations in challenging embodied and egocentric tasks demonstrate that strict domain alignment and high data quality are far more critical to performance than simple dataset scaling.
  arXiv  Detail & Related papers  (2026-05-26T17:59:20Z)
• HorizonBench: Long-Horizon Personalization with Evolving Preferences [73.32940850193681]
  We introduce a data generator that produces conversations from a structured mental state graph.<n>From it construct HorizonBench, a benchmark of 4,245 items from 360 simulated users with 6-month conversation histories.<n> HorizonBench provides a testbed for long-context modeling, memory-augmented architectures, theory-of-mind reasoning, and user modeling.
  arXiv  Detail & Related papers  (2026-04-19T06:55:10Z)
• Prediction Arena: Benchmarking AI Models on Real-World Prediction Markets [0.0]
  Prediction Arena is a benchmark for evaluating AI models' predictive accuracy and decision-making by enabling them to trade autonomously on live prediction markets with real capital.<n>We track two cohorts: six frontier models in live trading (Cohort 1, full period) and four next-generation models in paper trading (Cohort 2, 3-day preliminary)<n>Our analysis identifies a clear performance hierarchy: initial prediction accuracy and the ability to capitalize on correct predictions are the main drivers.
  arXiv  Detail & Related papers  (2026-03-28T06:13:17Z)
• How2Everything: Mining the Web for How-To Procedures to Evaluate and Improve LLMs [49.61011897610774]
  How2Everything is a framework to evaluate and improve goal-conditioned procedure generation.<n>Our framework includes How2Mine, which mines 351K procedures from 980K web pages across 14 topics.<n>How2Score is an evaluation protocol that uses an LLM judge to detect whether a generation contains any critical failure that would prevent achieving the goal.
  arXiv  Detail & Related papers  (2026-02-09T15:47:14Z)
• Intention-Conditioned Flow Occupancy Models [80.42634994902858]
  Large-scale pre-training has fundamentally changed how machine learning research is done today.<n>Applying this same framework to reinforcement learning is appealing because it offers compelling avenues for addressing core challenges in RL.<n>Recent advances in generative AI have provided new tools for modeling highly complex distributions.
  arXiv  Detail & Related papers  (2025-06-10T15:27:46Z)
• A data-driven framework for team selection in Fantasy Premier League [0.552480439325792]
  Under a fixed budget, managers select squads to maximize future Fantasy Premier League points.<n>This study formulates lineup selection as data-driven optimization.<n>Experiments on the 2023/24 Premier League season show that ARIMA with a constrained budget and a rolling window yields the most consistent out-of-sample performance.
  arXiv  Detail & Related papers  (2025-05-04T16:21:59Z)
• One Thousand and One Pairs: A "novel" challenge for long-context language models [56.60667988954638]
  NoCha is a dataset of 1,001 pairs of true and false claims about 67 fictional books. Our annotators confirm that the largest share of pairs in NoCha require global reasoning over the entire book to verify. On average, models perform much better on pairs that require only sentence-level retrieval vs. global reasoning.
  arXiv  Detail & Related papers  (2024-06-24T02:03:57Z)
• Evaluating Soccer Match Prediction Models: A Deep Learning Approach and Feature Optimization for Gradient-Boosted Trees [0.8009842832476994]
  The 2023 Soccer Prediction Challenge required the prediction of match results first in terms of the exact goals scored by each team, and second, in terms of the probabilities for a win, draw, and loss. A CatBoost model was employed using pi-ratings as the features, which were initially identified as the optimal choice for calculating the win/draw/loss probabilities. In this study, we aimed to assess the performance of a deep learning model and determine the optimal feature set for a gradient-boosted tree model.
  arXiv  Detail & Related papers  (2023-09-26T10:05:46Z)
• L-Eval: Instituting Standardized Evaluation for Long Context Language Models [91.05820785008527]
  We propose L-Eval to institute a more standardized evaluation for long context language models (LCLMs) We build a new evaluation suite containing 20 sub-tasks, 508 long documents, and over 2,000 human-labeled query-response pairs. Results show that popular n-gram matching metrics generally can not correlate well with human judgment.
  arXiv  Detail & Related papers  (2023-07-20T17:59:41Z)
• Using Experts' Opinions in Machine Learning Tasks [0.0]
  We propose a general three-step framework for utilizing experts' insights in machine learning tasks. For the case study, we have chosen the task of predicting NCAA Men's Basketball games, which has been the focus of a group of Kaggle competitions. Results suggest that the good performance and high scores of the past models are a result of chance, and not because of a good-performing and stable model.
  arXiv  Detail & Related papers  (2020-08-10T15:48:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.