FIFA World Cup 2026 as a Contamination-Free Benchmark for LLM Forecasting Agents: Four Models, a Bookmaker, and 104 Matches
- URL: http://arxiv.org/abs/2607.17765v1
- Date: Mon, 20 Jul 2026 10:00:11 GMT
- Title: FIFA World Cup 2026 as a Contamination-Free Benchmark for LLM Forecasting Agents: Four Models, a Bookmaker, and 104 Matches
- Abstract summary: We introduce WC2026-Agents, a benchmark and dataset for evaluating large language models (LLMs)<n>For every one of the 104 matches of the 2026 FIFA World Cup, four frontier models ran an identical search-act-reflect loop.<n>We pair the four agents with a fifth competitor drawn from the same information environment -- the pre-match betting market.
- Score: 2.8722431362974077
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: We introduce WC2026-Agents, a benchmark and dataset for evaluating large language models (LLMs) as autonomous forecasting agents on real, future events. For every one of the 104 matches of the 2026 FIFA World Cup, four frontier models -- Claude Opus 4.8, ChatGPT (GPT-5.5, high reasoning), Gemini 3.1 Pro, and Grok (Expert Mode) -- ran an identical search-act-reflect loop: gather evidence with a web tool, commit to a 1X2 (team-A win / draw / team-B win) distribution and a virtual 100-USD bet, and, after the match, reflect given only the final score. Because every match kicked off after the models' training cutoffs, the benchmark is contamination-free by construction. Crucially, we pair the four agents with a fifth competitor drawn from the same information environment -- the pre-match betting market -- collected as per-match 1X2 odds, giving an economically grounded baseline and letting us score not just what an agent predicts but what it does with money. The release contains 416 forecasts and 414 reflections with verbatim reasoning, ground truth (including penalty shootouts), odds, and a reproducible evaluation suite. A reference evaluation surfaces findings that raw accuracy hides: the four agents issue an identical top pick in 92% of matches and none beats the market's Brier score; indeed, a naive flat stake on the market favorite out-earns all four agents. Yet the agents diverge sharply as decision-makers: betting return-on-investment ranges from -18% to +10%, fading the market is unprofitable for all four, the share of forecasts that cite the market ranges from 12% to 100%, and self-reported error rates on wrong picks range from 36% to 86%. The benchmark thus measures calibration, decision quality, and self-knowledge -- axes on which frontier models differ even when their predictions do not. Data and code: https://github.com/graphuofm/FIFA2026LLM
Related papers
- Marginal Fidelity Does Not Establish User Simulation in Demographic Synthetic Survey Panels: Response Contracts, Support Collapse and Conditioning Failure [51.736723807086385]
Population-marginal agreement is evidence about an elicitation contract and an estimand obtainable without simulated respondents, not evidence of individual simulation.<n>On nine aligned model-battery pairs, a no-persona population-prevalence query averages 6.27 MAE versus 12.39 for committed panels and wins all nine comparisons.
arXiv Detail & Related papers (2026-09-07T10:22:22Z) - FM-Bench: A Benchmark for Long-Horizon Management with Competing Agents [16.48804532565337]
FM-Bench (Football Management Benchmark) measures bounded tasks.<n>An agent runs a football club for 20 in-game years through 26 tools and roughly 340 to 400 decision stops.<n>It drafts a squad on the same budget as every rival, trades players, negotiates contracts, invests in facilities and youth, sets lineups, and answers to a board that can fire it.
arXiv Detail & Related papers (2026-08-19T01:33:36Z) - When LLMs Agree, Are They Right? Auditing Self-Consistency and Cross-Model Agreement as Confidence Signals [0.0]
LLM-as-judge is increasingly the default for evaluating AI systems in enterprise pipelines.<n>We show that consistency -- agreement among judges, or a model's own samples -- indicates correctness.<n>We ask when agreement is nonetheless a usable proxy, in a large-scale cross-runner study.
arXiv Detail & Related papers (2026-07-09T02:46:51Z) - Knowledge Index of Noah's Ark [63.143852586221534]
We introduce KINA, an 899-item benchmark across 261 fine-grained disciplines.<n>We prove a bonus-on-bar tournament weakly FOSD-dominates flat payment in released-review quality.<n>The top model, Gemini-3.1-Pro-Preview, reaches 53.17%, followed by Claude-Opus-4.6 at 49.92% and GPT-5.4 at 48.55%.
arXiv Detail & Related papers (2026-06-03T17:06:49Z) - Evaluating Commercial AI Chatbots as News Intermediaries [85.32040752972836]
Best systems achieve over 90% multiple-choice accuracy on questions about events reported hours earlier.<n>Every model achieves its lowest accuracy on Hindi.<n> retrieval, not reasoning, failures drive over 70% of all errors.
arXiv Detail & Related papers (2026-05-21T17:42:07Z) - Soohak: A Mathematician-Curated Benchmark for Evaluating Research-level Math Capabilities of LLMs [86.49905745865038]
Soohak is a 439-problem benchmark newly authored from scratch by 64 mathematicians.<n>The dataset will be publicly released in late 2026, with model evaluations available upon request in the interim.
arXiv Detail & Related papers (2026-05-09T17:14:22Z) - How Far Is Document Parsing from Solved? PureDocBench: A Source-TraceableBenchmark across Clean, Degraded, and Real-World Settings [56.70440596502351]
The past year has seen over 20 open document parsing models, yet benchmarks almost exclusively on OmniDocBench.<n>We present PureDocBench, a benchmark thatrenders document images from HTML/CSS, covering 10 domains, 66 pages, and 1,475 pages.
arXiv Detail & Related papers (2026-05-08T09:30:31Z) - Foresight Arena: An On-Chain Benchmark for Evaluating AI Forecasting Agents [0.0]
We introduce Foresight Arena, the first permissionless, on-chain benchmark for evaluating AI forecasting agents on real-world prediction markets.<n>Performance is measured by the Brier Score and a novel Alpha Score -- proper scoring rules that incentivize honest probability reporting.<n>We show that detecting a true edge of $* = 0.02$ at 80% power requires approximately 350 resolved binary predictions.
arXiv Detail & Related papers (2026-05-01T05:33:10Z) - Prediction Arena: Benchmarking AI Models on Real-World Prediction Markets [0.0]
Prediction Arena is a benchmark for evaluating AI models' predictive accuracy and decision-making by enabling them to trade autonomously on live prediction markets with real capital.<n>We track two cohorts: six frontier models in live trading (Cohort 1, full period) and four next-generation models in paper trading (Cohort 2, 3-day preliminary)<n>Our analysis identifies a clear performance hierarchy: initial prediction accuracy and the ability to capitalize on correct predictions are the main drivers.
arXiv Detail & Related papers (2026-03-28T06:13:17Z) - Going All-In on LLM Accuracy: Fake Prediction Markets, Real Confidence Signals [0.0]
We generated 100 math and logic questions with verifiable answers.<n>Three Predictor models then forecasted, for each question-baseline pair, if the baseline would answer correctly.<n>Across 5,400 predictions per condition, Incentive runs showed modestly higher accuracy.<n>"Whale" bets of 40,000+ coins were correct 99% of the time, while small bets (1,000 coins) showed only 74% accuracy.
arXiv Detail & Related papers (2025-12-01T19:04:25Z) - Shoot First, Ask Questions Later? Building Rational Agents that Explore and Act Like People [81.63702981397408]
Given limited resources, to what extent do agents based on language models (LMs) act rationally?<n>We develop methods to benchmark and enhance agentic information-seeking, drawing on insights from human behavior.<n>For Spotter agents, our approach boosts accuracy by up to 14.7% absolute over LM-only baselines; for Captain agents, it raises expected information gain (EIG) by up to 0.227 bits (94.2% of the achievable noise ceiling)
arXiv Detail & Related papers (2025-10-23T17:57:28Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.