Prediction Arena: Benchmarking AI Models on Real-World Prediction Markets

• URL: http://arxiv.org/abs/2604.07355v1
• Date: Sat, 28 Mar 2026 06:13:17 GMT
• Title: Prediction Arena: Benchmarking AI Models on Real-World Prediction Markets
• Abstract summary: Prediction Arena is a benchmark for evaluating AI models' predictive accuracy and decision-making by enabling them to trade autonomously on live prediction markets with real capital.<n>We track two cohorts: six frontier models in live trading (Cohort 1, full period) and four next-generation models in paper trading (Cohort 2, 3-day preliminary)<n>Our analysis identifies a clear performance hierarchy: initial prediction accuracy and the ability to capitalize on correct predictions are the main drivers.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce Prediction Arena, a benchmark for evaluating AI models' predictive accuracy and decision-making by enabling them to trade autonomously on live prediction markets with real capital. Unlike synthetic benchmarks, Prediction Arena tests models in environments where trades execute on actual exchanges (Kalshi and Polymarket), providing objective ground truth that cannot be gamed or overfitted. Each model operates as an independent agent starting with $10,000, making autonomous decisions every 15-45 minutes. Over a 57-day longitudinal evaluation (January 12 to March 9, 2026), we track two cohorts: six frontier models in live trading (Cohort 1, full period) and four next-generation models in paper trading (Cohort 2, 3-day preliminary). For Cohort 1, final Kalshi returns range from -16.0% to -30.8%. Our analysis identifies a clear performance hierarchy: initial prediction accuracy and the ability to capitalize on correct predictions are the main drivers, while research volume shows no correlation with outcomes. A striking cross-platform contrast emerges from parallel Polymarket live trading: Cohort 1 models averaged only -1.1% on Polymarket vs. -22.6% on Kalshi, with grok-4-20-checkpoint achieving a 71.4% settlement win rate - the highest across any platform or cohort. gemini-3.1-pro-preview (Cohort 2), which executed zero trades on Kalshi, achieved +6.02% on Polymarket in 3 days - the best return of any model across either cohort - demonstrating that platform design has a profound effect on which models succeed. Beyond performance, we analyze computational efficiency (token usage, cycle time), settlement accuracy, exit patterns, and market preferences, providing a comprehensive view of how frontier models behave under real financial pressure.

Related papers

• Marginal Fidelity Does Not Establish User Simulation in Demographic Synthetic Survey Panels: Response Contracts, Support Collapse and Conditioning Failure [51.736723807086385]
  Population-marginal agreement is evidence about an elicitation contract and an estimand obtainable without simulated respondents, not evidence of individual simulation.<n>On nine aligned model-battery pairs, a no-persona population-prevalence query averages 6.27 MAE versus 12.39 for committed panels and wins all nine comparisons.
  arXiv  Detail & Related papers  (2026-09-07T10:22:22Z)
• One Capability or Many? Testing the Economic Validity of Frontier AI Evaluation [0.0]
  We test whether benchmarks measure capability distinct from general test-taking.<n>Economic benchmarks add incremental predictive information to a largely date-driven general factor.<n> Leaderboards remain a sound guide to overall progress, but most of the gap between models released months apart is calendar.
  arXiv  Detail & Related papers  (2026-08-29T19:55:20Z)
• Design and Evaluation of Multi-Agent AI Oracle Systems for Prediction Market Resolution [0.0]
  Prediction markets aggregate collective intelligence to forecast uncertain events.<n>Existing oracle systems tradeoff fast but brittle automation against accurate but costly human arbitration.<n>We evaluate whether multi-agent LLM architectures can improve oracle resolution accuracy over single-model baselines.
  arXiv  Detail & Related papers  (2026-05-29T03:44:19Z)
• Putting HUMANS first: Efficient LAM Evaluation with Human Preference Alignment [53.72927532626824]
  We show that subsets of just 50 examples (0.3% of data) can achieve over 0.93 Pearson correlation with full benchmark scores.<n>To better predict preferences, we trained regression models on these selected subsets, achieving 0.98 correlation.<n>This demonstrates that in regression modeling, well-curated subsets outpredict the full benchmark, showing quality over quantity.
  arXiv  Detail & Related papers  (2026-04-20T00:57:31Z)
• TimeSeek: Temporal Reliability of Agentic Forecasters [0.5352699766206807]
  Models are most competitive early in a market's life and on high-uncertainty markets, but much less competitive near resolution and on strong-consensus markets.<n>Web search improves pooled Brier Skill Score (BSS) for every model overall, yet hurts in 12% of model-checkpoint pairs.<n>Simple two-model ensembles reduce error without surpassing the market overall.
  arXiv  Detail & Related papers  (2026-04-05T18:38:39Z)
• Stock Market Prediction Using Node Transformer Architecture Integrated with BERT Sentiment Analysis [1.2362187555287152]
  This paper presents an integrated framework combining a node transformer architecture with BERT-based sentiment analysis for stock price forecasting.<n>The proposed model represents the stock market as a graph structure where individual stocks form nodes and edges capture relationships.<n> Experiments on 20 S&P 500 stocks spanning January 1982 to March 2025 demonstrate that the integrated model achieves a mean absolute percentage error (MAPE) of 0.80% for one-day-ahead predictions.
  arXiv  Detail & Related papers  (2026-03-06T05:15:22Z)
• Hybrid Quantum-Classical Ensemble Learning for S\&P 500 Directional Prediction [0.2538209532048867]
  We introduce a hybrid ensemble framework combining quantum sentiment analysis, Decision Transformer architecture, and strategic model selection.<n>We achieve 60.14% directional accuracy on S&P 500 prediction, a 3.10% improvement over individual models.
  arXiv  Detail & Related papers  (2025-12-06T22:22:09Z)
• Forecasting Frontier Language Model Agent Capabilities [0.7499722271664147]
  We evaluate six forecasting methods that predict downstream capabilities of Language Models (LMs)<n>We use "one-step" approaches that predict benchmark scores from input metrics like compute or model release date directly or "two-step" approaches that first predict an intermediate metric like the principal component of cross-benchmark performance (PC-1) and human-evaluated competitive Elo ratings.<n>Our forecast predicts that by the beginning of 2026, non-specialized LM agents with low capability elicitation will reach a success rate of 54% on SWE-Bench Verified, while state-of-the-art LM agents will reach an 87% success rate.
  arXiv  Detail & Related papers  (2025-02-21T02:34:17Z)
• Improved Bayes Risk Can Yield Reduced Social Welfare Under Competition [99.7047087527422]
  In this work, we demonstrate that competition can fundamentally alter the behavior of machine learning scaling trends. We find many settings where improving data representation quality decreases the overall predictive accuracy across users. At a conceptual level, our work suggests that favorable scaling trends for individual model-providers need not translate to downstream improvements in social welfare.
  arXiv  Detail & Related papers  (2023-06-26T13:06:34Z)
• Stock Price Prediction Under Anomalous Circumstances [81.37657557441649]
  This paper aims to capture the movement pattern of stock prices under anomalous circumstances. We train ARIMA and LSTM models at the single-stock level, industry level, and general market level. Based on 100 companies' stock prices in the period of 2016 to 2020, the models achieve an average prediction accuracy of 98%.
  arXiv  Detail & Related papers  (2021-09-14T18:50:38Z)
• Test-time Collective Prediction [73.74982509510961]
  Multiple parties in machine learning want to jointly make predictions on future test points. Agents wish to benefit from the collective expertise of the full set of agents, but may not be willing to release their data or model parameters. We explore a decentralized mechanism to make collective predictions at test time, leveraging each agent's pre-trained model.
  arXiv  Detail & Related papers  (2021-06-22T18:29:58Z)
• Feature Learning for Stock Price Prediction Shows a Significant Role of Analyst Rating [0.38073142980733]
  A set of 5 technical indicators and 23 fundamental indicators was identified to establish the possibility of generating excess returns on the stock market. From any given day, we were able to predict the direction of change in price by 1% up to 10 days in the future. The predictions had an overall accuracy of 83.62% with a precision of 85% for buy signals and a recall of 100% for sell signals.
  arXiv  Detail & Related papers  (2021-03-13T03:56:29Z)
• Characterizing Fairness Over the Set of Good Models Under Selective Labels [69.64662540443162]
  We develop a framework for characterizing predictive fairness properties over the set of models that deliver similar overall performance. We provide tractable algorithms to compute the range of attainable group-level predictive disparities. We extend our framework to address the empirically relevant challenge of selectively labelled data.
  arXiv  Detail & Related papers  (2021-01-02T02:11:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.