WorldCup Arena: Prospective, Leakage-Free Evaluation of Frontier LLMs on a Live Tournament

• URL: http://arxiv.org/abs/2608.04008v1
• Date: Tue, 04 Aug 2026 17:59:55 GMT
• Title: WorldCup Arena: Prospective, Leakage-Free Evaluation of Frontier LLMs on a Live Tournament
• Abstract summary: Six frontier LLMs were asked to fill in a seven-market prediction card for all 104 matches.<n>No answer existed when the question was asked, so the evaluation is leakage-free by construction.<n>What the tournament establishes is a set of behaviours the six systems share.
• Score: 1.4943316571343976
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Benchmarks that measure the forecasting ability of large language models are almost always retrospective: the event has happened, the answer is somewhere on the Web, and the evaluation must defend itself against memorisation. We report the opposite design. Over the 39 days of the 2026 FIFA World Cup, six frontier LLMs -- all with extended thinking and native server-side web search -- were asked before every kickoff, one match at a time, to fill in a seven-market prediction card for all 104 matches, plus 12 group winners and a pre-tournament outright pool; no answer existed when the question was asked, so the evaluation is leakage-free by construction rather than by filtering, and the frozen archive holds 4,494 scored predictions. What the tournament establishes is a set of behaviours the six systems share. On match outcome they average 63.9%, level with backing the bookmaker's favourite -- which is in fact what they usually do. They agree with one another far more often than they are right, so a majority vote adds nothing. They under-commit to draws and to goals, and crowd their scoreline picks onto a single prototypical result. Accuracy tracks how lopsided a fixture is rather than how much is known about it: it collapses in the closest ties, where the dossiers are richest, while questions about the tournament as a whole are answered well. On this task the current generation of frontier systems is not sharply differentiated: the standings hold up at the top and the bottom across the run and churn in the middle, and the margins stay narrow throughout. The briefing dossiers, fixtures and official results are released as a benchmark, together with the scoring code.

Related papers

• DraftFM: A FoundationModel for Day-Zero Drafting in Magic: The Gathering [0.0]
  DraftFM scores every card available in the current pack conditioned on the drafted pool and the state of the draft.<n>A network fitted on 149 million human picks from 29 expansions predicts held-out picks in three expansions in their entirety.<n>The same architecture produced a card ranking for the then-unreleased set The Hobbit, sealed with its complete cryptographic provenance.
  arXiv  Detail & Related papers  (2026-08-20T02:13:23Z)
• FM-Bench: A Benchmark for Long-Horizon Management with Competing Agents [16.48804532565337]
  FM-Bench (Football Management Benchmark) measures bounded tasks.<n>An agent runs a football club for 20 in-game years through 26 tools and roughly 340 to 400 decision stops.<n>It drafts a squad on the same budget as every rival, trades players, negotiates contracts, invests in facilities and youth, sets lineups, and answers to a board that can fire it.
  arXiv  Detail & Related papers  (2026-08-19T01:33:36Z)
• AI World Cup 2026: Benchmarking Large Language Models for End-to-End Football Tournament Prediction [0.0]
  This paper reports the completed emphAI World Cup benchmark, in which ten-based assistants made a single pre-tournament forecast of the entire 2026 FIFA World Cup.<n>Forecasts covered group-stage scores, group rankings, the knockout bracket, final placings, confidence values, and short explanations.<n>GPT-5.5 Thinking finished first with 744 points, followed by GPT-5.5 with 717, Gemini with 699, and Qwen 3.7 with 687.
  arXiv  Detail & Related papers  (2026-08-04T10:07:14Z)
• WorldCupArena: Fine-Grained Evaluation of Language Models and Deep-Research Agents on Football Forecasting [10.941871398755403]
  We present WorldCupArena, a benchmark for language models and deep-research agents.<n>It predicts the result and score, likely players and events, match statistics, and the outcome of the competition.<n>We report result accuracy, exact-score accuracy, and a scoreline score that gives some credit when a predicted score is close but not exact.
  arXiv  Detail & Related papers  (2026-07-20T15:52:48Z)
• FIFA World Cup 2026 as a Contamination-Free Benchmark for LLM Forecasting Agents: Four Models, a Bookmaker, and 104 Matches [2.8722431362974077]
  We introduce WC2026-Agents, a benchmark and dataset for evaluating large language models (LLMs)<n>For every one of the 104 matches of the 2026 FIFA World Cup, four frontier models ran an identical search-act-reflect loop.<n>We pair the four agents with a fifth competitor drawn from the same information environment -- the pre-match betting market.
  arXiv  Detail & Related papers  (2026-07-20T10:00:11Z)
• Training-Free Off-Screen Player Imputation for Broadcast-Based Spatial Football Analytics [0.9185901163605424]
  Spatial football metrics assume access to the positions of all 22 players.<n>Broadcast main camera shows only 10-16 of them at any moment.<n>We quantify the resulting distortion with an open, reproducible benchmark.
  arXiv  Detail & Related papers  (2026-07-13T13:33:45Z)
• Reasoning Arena: Trace Tournaments When Verifiable Rewards Fall Short [51.667769734342635]
  Reinforcement learning with verifiable rewards (RLVR) has become a leading paradigm for improving the reasoning ability of large language models.<n>We propose Reasoning Arena, an adaptive training framework that routes such non-diverse reward groups to a judge system instead of discarding them.<n>We show that Reasoning Arena consistently outperforms the RLVR baseline by 7.6% on average in competition mathematics and coding benchmarks.
  arXiv  Detail & Related papers  (2026-06-08T11:57:17Z)
• OVO-S-Bench: A Hierarchical Benchmark for Streaming Spatial Intelligence in Multimodal LLMs [49.82435858489898]
  OVOS-Bench is a fully-annotated benchmark for streaming human-centric spatial intelligence.<n>It consists of 1,680 questions on 348 source videos.<n>Each question carries a query and an evidence interval, and at evaluation, the model sees only the prefix preceding the query.
  arXiv  Detail & Related papers  (2026-06-02T16:51:32Z)
• MINDGAMES: A Live Arena for Evaluating Social and Strategic Reasoning in Multi-Agent LLMs [54.81359054218573]
  We introduce Mindgames, a multi-game arena and evaluation platform for large language models (LLMs)<n>Mindgames provides a unified interaction interface, TrueSkill-based rating, and full trajectory logging across four game environments.<n>We analyze 29,571 multi-agent games with turn-level observations, actions, and rewards, together with MG-Ref, a deterministic offline tournament protocol.
  arXiv  Detail & Related papers  (2026-05-28T07:33:47Z)
• OpenDeepThink: Parallel Reasoning via Bradley-Terry Aggregation [53.88666485159289]
  We introduce OpenDeepThink, a population-based test-time compute framework that selects via pairwise Bradley-Terry comparison.<n>OpenDeepThink raises Gemini 3.1 Pro's effective Codeforces Elo by +405 points in eight sequential LLM-call rounds.<n>We release CF-73, a curated set of 73 expert-rated Codeforces problems with International Grandmaster annotation and 99% local-evaluation agreement against the official verdict.
  arXiv  Detail & Related papers  (2026-05-14T17:57:40Z)
• Results and Retrospective Analysis of the CODS 2025 AssetOpsBench Challenge [3.5112866130906926]
  We revisit the CODS 2025 assetopslive challenge, a privacy-aware Codabench competition on industrial multi-agent orchestration.<n>We combine final rank sheets, a 300- server log, 149-team registrations, best-submission exports, the organizer winners report, the companion assetopslive system paper, and verified planning-track source trees.<n> hidden evaluation changes the story: public and private scores correlate moderately in planning but negatively in execution.
  arXiv  Detail & Related papers  (2026-05-08T22:00:58Z)
• CodeClash: Benchmarking Goal-Oriented Software Engineering [63.65464283837602]
  We run 1680 tournaments (25,200 rounds total) to evaluate 8 LMs across 6 arenas.<n>Our results reveal that while models exhibit diverse development styles, they share fundamental limitations in strategic reasoning.<n>We open-source CodeClash to advance the study of autonomous, goal-oriented code development.
  arXiv  Detail & Related papers  (2025-11-02T07:42:51Z)
• The Leaderboard Illusion [61.27964089648608]
  Arena has emerged as the go-to leaderboard for ranking the most capable AI systems.<n>We identify systematic issues that have resulted in a distorted playing field.
  arXiv  Detail & Related papers  (2025-04-29T15:48:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.