PolyBench: Benchmarking LLM Forecasting and Trading Capabilities on Live Prediction Market Data

• URL: http://arxiv.org/abs/2604.14199v1
• Date: Fri, 03 Apr 2026 06:25:21 GMT
• Title: PolyBench: Benchmarking LLM Forecasting and Trading Capabilities on Live Prediction Market Data
• Abstract summary: We present textbfPolyBench,' a benchmark for predicting real-world events from live market signals.<n>Using PolyBench, we evaluate seven state-of-the-art Large Language Models.<n>Our framework assesses directional accuracy, our proposed Confidence-Weighted Return (CWR), Annualized Percentage Yield (APY), and Sharpe ratio.
• Score: 3.7753883100108525
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Predicting real-world events from live market signals demands systems that fuse qualitative news with quantitative order-book dynamics under strict temporal discipline -- a challenge existing benchmarks fail to capture. We present \textbf{PolyBench}, a multimodal benchmark derived from Polymarket that records point-in-time cross-sections of 38,666 binary prediction markets spanning 4,997 events, synchronously coupling each snapshot with a Central Limit Order Book (CLOB) state and a real-time news stream. Using PolyBench, we evaluate seven state-of-the-art Large Language Models -- spanning open- and closed-source families -- generating 36,165 predictions under identical, timestamp-locked market states collected between February 6 and 12, 2026. Our multidimensional framework assesses directional accuracy, our proposed Confidence-Weighted Return (CWR), Annualized Percentage Yield (APY), and Sharpe ratio via realistic order-book execution simulation. The results reveal a pronounced performance divergence: only two of seven models achieve positive financial returns -- MiMo-V2-Flash at \textbf{17.6%} CWR and Gemini-3-Flash at 6.2% CWR -- while the remaining five incur losses despite uniformly high stated confidence. These findings highlight the gap between surface-level language fluency and genuine probabilistic reasoning under live market uncertainty, and establish PolyBench as a contamination-proof, financially-grounded evaluation standard for future LLM research. Our dataset and code available at \underline{\href{https://github.com/PolyBench/PolyBench}{https://github.com/PolyBench/PolyBench}}.

Related papers

• BiasMix-Finance: Post-Generation KYC Guardrails for LLM Portfolio Advice [4.108510562556002]
  Large language models (LLMs) can generate plausible-sounding ETF portfolios while silently violating basic constraints on risk, fees, and diversification.<n>We study a model-agnostic, asset-agnostic post-generation guardrail pipeline.<n>We introduce BiasMix-Finance (Mini), a compact stress-test benchmark for constrained decision-making under biased generations.
  arXiv  Detail & Related papers  (2026-08-14T05:32:14Z)
• PortBench: A Correlation-Aware, Full-Pipeline Benchmark for LLM-Driven Portfolio Management [15.684384084836223]
  Large language models (LLMs) have shown strong performance across diverse financial tasks, yet portfolio management (PM) remains poorly benchmarked.<n>We introduce PortBench, a benchmark spanning six heterogeneous asset classes over ten years.<n>We find that despite strong performance on static financial QA, 90% of model-profile combinations fail to outperform a basic equal-weight allocation.
  arXiv  Detail & Related papers  (2026-05-27T03:08:23Z)
• PolySwarm: A Multi-Agent Large Language Model Framework for Prediction Market Trading and Latency Arbitrage [0.0]
  PolySwarm deploys a swarm of 50 diverse LLM personas that concurrently evaluate binary outcome markets.<n>A latency arbitrage module exploits stale Polymarket prices by deriving CEX-implied probabilities from a log-normal pricing model.
  arXiv  Detail & Related papers  (2026-04-04T22:51:06Z)
• QuitoBench: A High-Quality Open Time Series Forecasting Benchmark [32.21290355342465]
  Time series forecasting is critical across finance, healthcare, and cloud computing.<n>We introduce textscQuitoBench, a regime-balanced benchmark for time series forecasting.
  arXiv  Detail & Related papers  (2026-03-27T02:24:34Z)
• Forecasting Future Language: Context Design for Mention Markets [81.25011140991566]
  We study how input context should be designed to support accurate prediction in mention markets.<n>We find three insights: (1) richer context consistently improves forecasting performance; (2) market-conditioned prompting (MCP) treats the market probability as a prior and updates it using textual evidence, yields better-calibrated forecasts; and (3) a mixture of the market probability and MCP (MixMCP) outperforms the market baseline.
  arXiv  Detail & Related papers  (2026-02-04T12:43:31Z)
• LiveTradeBench: Seeking Real-World Alpha with Large Language Models [26.976122048323873]
  Large language models (LLMs) achieve strong performance across benchmarks.<n>These tests occur in static settings, lacking real dynamics and uncertainty.<n>We introduce LiveTradeBench, a live trading environment for evaluating LLM agents in realistic and evolving markets.
  arXiv  Detail & Related papers  (2025-11-05T16:47:26Z)
• Agentic World Modeling for 6G: Near-Real-Time Generative State-Space Reasoning [70.56067503630486]
  We argue that sixth-generation (6G) intelligence is not fluent token prediction but calibrated the capacity to imagine and choose.<n>We show that WM-MS3M cuts mean absolute error (MAE) by 1.69% versus MS3M with 32% fewer parameters and similar latency, and achieves 35-80% lower root mean squared error (RMSE) than attention/hybrid baselines with 2.3-4.1x faster inference.
  arXiv  Detail & Related papers  (2025-11-04T17:22:22Z)
• Increase Alpha: Performance and Risk of an AI-Driven Trading Framework [0.0]
  There are inefficiencies in financial markets, with unexploited patterns in price, volume, and cross-sectional relationships.<n>At Increase Alpha, we built a deep-learning framework that maps over 800 U.S. equities into daily directional signals.<n>We evaluate its real-time performance through transparent, industry standard metrics.
  arXiv  Detail & Related papers  (2025-09-20T14:37:02Z)
• FinTSB: A Comprehensive and Practical Benchmark for Financial Time Series Forecasting [58.70072722290475]
  Financial time series (FinTS) record the behavior of human-brain-augmented decision-making.<n>FinTSB is a comprehensive and practical benchmark for financial time series forecasting.
  arXiv  Detail & Related papers  (2025-02-26T05:19:16Z)
• Exploring Response Uncertainty in MLLMs: An Empirical Evaluation under Misleading Scenarios [49.53589774730807]
  Multimodal large language models (MLLMs) have recently achieved state-of-the-art performance on tasks ranging from visual question answering to video understanding.<n>We reveal a response uncertainty phenomenon: twelve state-of-the-art open-source MLLMs overturn a previously correct answer in 65% of cases after receiving a single deceptive cue.
  arXiv  Detail & Related papers  (2024-11-05T01:11:28Z)
• Diffusion Variational Autoencoder for Tackling Stochasticity in Multi-Step Regression Stock Price Prediction [54.21695754082441]
  Multi-step stock price prediction over a long-term horizon is crucial for forecasting its volatility. Current solutions to multi-step stock price prediction are mostly designed for single-step, classification-based predictions. We combine a deep hierarchical variational-autoencoder (VAE) and diffusion probabilistic techniques to do seq2seq stock prediction. Our model is shown to outperform state-of-the-art solutions in terms of its prediction accuracy and variance.
  arXiv  Detail & Related papers  (2023-08-18T16:21:15Z)
• Multi-Asset Spot and Option Market Simulation [52.77024349608834]
  We construct realistic spot and equity option market simulators for a single underlying on the basis of normalizing flows. We leverage the conditional invertibility property of normalizing flows and introduce a scalable method to calibrate the joint distribution of a set of independent simulators.
  arXiv  Detail & Related papers  (2021-12-13T17:34:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.