TimeSeek: Temporal Reliability of Agentic Forecasters

• URL: http://arxiv.org/abs/2604.04220v1
• Date: Sun, 05 Apr 2026 18:38:39 GMT
• Title: TimeSeek: Temporal Reliability of Agentic Forecasters
• Abstract summary: Models are most competitive early in a market's life and on high-uncertainty markets, but much less competitive near resolution and on strong-consensus markets.<n>Web search improves pooled Brier Skill Score (BSS) for every model overall, yet hurts in 12% of model-checkpoint pairs.<n>Simple two-model ensembles reduce error without surpassing the market overall.
• Score: 0.5352699766206807
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce TimeSeek, a benchmark for studying how the reliability of agentic LLM forecasters changes over a prediction market's lifecycle. We evaluate 10 frontier models on 150 CFTC-regulated Kalshi binary markets at five temporal checkpoints, with and without web search, for 15,000 forecasts total. Models are most competitive early in a market's life and on high-uncertainty markets, but much less competitive near resolution and on strong-consensus markets. Web search improves pooled Brier Skill Score (BSS) for every model overall, yet hurts in 12% of model-checkpoint pairs, indicating that retrieval is helpful on average but not uniformly so. Simple two-model ensembles reduce error without surpassing the market overall. These descriptive results motivate time-aware evaluation and selective-deference policies rather than a single market snapshot or a uniform tool-use setting.

Related papers

• Dynamically Allocating Evaluation Effort for Model Ranking [60.00410111594536]
  We formalize multi-model human evaluation as a best-arm identification problem in a multi-armed bandit setup with correlated arms.<n>We show that it improves discrimination between top-performing models.<n>This makes evaluations faster, cheaper and more aligned with large-scale competition evaluation goals.
  arXiv  Detail & Related papers  (2026-08-04T10:33:04Z)
• StakeBench: Evaluating Language Understanding Grounded in Market Commitment [5.902169076249955]
  StakeBench is an evaluation framework for language understanding grounded in market commitment.<n> StakeBench links 560,876 comments from 2,261 resolved markets to verified position, action, and market-odds records.<n>Four diagnostic tasks test whether models detect market commitment, identify the revealed side, anticipate future action, and perform collective odds projection.
  arXiv  Detail & Related papers  (2026-05-25T17:38:30Z)
• Empirical evaluation of Time Series Foundation Models for Day-ahead and Imbalance Electricity Price Forecasting in Belgium [0.0]
  Time Series Foundation Models (TSFMs) promise zero-shot forecasting capabilities with minimal task-specific training.<n>This study provides a systematic empirical evaluation of several TSFMs for forecasting Belgian day-ahead and imbalance electricity prices.<n>We find that TSFMs exhibit genuine zero-shot forecasting skills but still struggle under extreme market conditions.
  arXiv  Detail & Related papers  (2026-05-16T15:30:23Z)
• Foresight Arena: An On-Chain Benchmark for Evaluating AI Forecasting Agents [0.0]
  We introduce Foresight Arena, the first permissionless, on-chain benchmark for evaluating AI forecasting agents on real-world prediction markets.<n>Performance is measured by the Brier Score and a novel Alpha Score -- proper scoring rules that incentivize honest probability reporting.<n>We show that detecting a true edge of $* = 0.02$ at 80% power requires approximately 350 resolved binary predictions.
  arXiv  Detail & Related papers  (2026-05-01T05:33:10Z)
• PolySwarm: A Multi-Agent Large Language Model Framework for Prediction Market Trading and Latency Arbitrage [0.0]
  PolySwarm deploys a swarm of 50 diverse LLM personas that concurrently evaluate binary outcome markets.<n>A latency arbitrage module exploits stale Polymarket prices by deriving CEX-implied probabilities from a log-normal pricing model.
  arXiv  Detail & Related papers  (2026-04-04T22:51:06Z)
• Forecasting Future Language: Context Design for Mention Markets [81.25011140991566]
  We study how input context should be designed to support accurate prediction in mention markets.<n>We find three insights: (1) richer context consistently improves forecasting performance; (2) market-conditioned prompting (MCP) treats the market probability as a prior and updates it using textual evidence, yields better-calibrated forecasts; and (3) a mixture of the market probability and MCP (MixMCP) outperforms the market baseline.
  arXiv  Detail & Related papers  (2026-02-04T12:43:31Z)
• Increase Alpha: Performance and Risk of an AI-Driven Trading Framework [0.0]
  There are inefficiencies in financial markets, with unexploited patterns in price, volume, and cross-sectional relationships.<n>At Increase Alpha, we built a deep-learning framework that maps over 800 U.S. equities into daily directional signals.<n>We evaluate its real-time performance through transparent, industry standard metrics.
  arXiv  Detail & Related papers  (2025-09-20T14:37:02Z)
• The NazoNazo Benchmark: A Cost-Effective and Extensible Test of Insight-Based Reasoning in LLMs [3.9977256267361754]
  We present Nazonazo, a cost-effective benchmark built from Japanese children's riddles to test insight-based reasoning.<n>No model except for GPT-5 is comparable to human performance, which achieves a 52.9% mean accuracy.
  arXiv  Detail & Related papers  (2025-09-18T07:50:04Z)
• Your AI, Not Your View: The Bias of LLMs in Investment Analysis [62.388554963415906]
  In finance, Large Language Models (LLMs) face frequent knowledge conflicts arising from discrepancies between their pre-trained parametric knowledge and real-time market data.<n>These conflicts are especially problematic in real-world investment services, where a model's inherent biases can misalign with institutional objectives.<n>We propose an experimental framework to investigate emergent behaviors in such conflict scenarios, offering a quantitative analysis of bias in investment analysis.
  arXiv  Detail & Related papers  (2025-07-28T16:09:38Z)
• Consistency Checks for Language Model Forecasters [54.62507816753479]
  We measure the performance of forecasters in terms of the consistency of their predictions on different logically-related questions.<n>We build an automated evaluation system that generates a set of base questions, instantiates consistency checks from these questions, elicits predictions of the forecaster, and measures the consistency of the predictions.
  arXiv  Detail & Related papers  (2024-12-24T16:51:35Z)
• Electricity Price Forecasting in the Irish Balancing Market [0.0]
  This work applies to the Irish balancing market a variety of price prediction techniques proven successful in the widely studied day-ahead market. We compare statistical, machine learning, and deep learning models using a framework that investigates the impact of different training sizes. An extensive numerical study shows that well-performing models in the day-ahead market do not perform well in the balancing one.
  arXiv  Detail & Related papers  (2024-02-09T15:18:00Z)
• Improved Bayes Risk Can Yield Reduced Social Welfare Under Competition [99.7047087527422]
  In this work, we demonstrate that competition can fundamentally alter the behavior of machine learning scaling trends. We find many settings where improving data representation quality decreases the overall predictive accuracy across users. At a conceptual level, our work suggests that favorable scaling trends for individual model-providers need not translate to downstream improvements in social welfare.
  arXiv  Detail & Related papers  (2023-06-26T13:06:34Z)
• Stock Price Prediction Under Anomalous Circumstances [81.37657557441649]
  This paper aims to capture the movement pattern of stock prices under anomalous circumstances. We train ARIMA and LSTM models at the single-stock level, industry level, and general market level. Based on 100 companies' stock prices in the period of 2016 to 2020, the models achieve an average prediction accuracy of 98%.
  arXiv  Detail & Related papers  (2021-09-14T18:50:38Z)
• Models, Markets, and the Forecasting of Elections [3.8138805042090325]
  We find systematic differences in accuracy over time, with markets performing better several months before the election, and the model performing better as the election approached. We propose a market design that incorporates model forecasts via a trading bot to generate synthetic predictions.
  arXiv  Detail & Related papers  (2021-02-06T19:05:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.