ForecastBench-Sim: A Simulated-World Forecasting Benchmark

• URL: http://arxiv.org/abs/2606.18686v1
• Date: Wed, 17 Jun 2026 04:52:41 GMT
• Title: ForecastBench-Sim: A Simulated-World Forecasting Benchmark
• Abstract summary: ForecastBench-Sim is a simulated-world forecasting benchmark built on game rollouts from Freeciv.<n>Forecasters receive a fixed world report and answer questions about hidden future states.<n>Because the world is simulated, the same setup can generate continuous or binary forecasting questions.
• Score: 11.4807141145402
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Forecasting benchmarks for general-purpose AI systems usually inherit the constraints of the real world: outcomes resolve slowly, tail events are rare, and counterfactual questions are difficult to score. We introduce ForecastBench-Sim, a simulated-world forecasting benchmark built on game rollouts from Freeciv, a turn-based strategy game modelled on the Civilization series. Forecasters receive a fixed world report (a structured snapshot of the current game state) and answer questions about hidden future states; the benchmark then continues the simulation and scores forecasts. Because the world is simulated, the same setup can generate continuous or binary forecasting questions at arbitrary time horizons, paired intervention worlds for conditional or causal questions, and resolved examples of rare or disruptive outcomes. We describe the benchmark pipeline, question families, scoring protocol, and release artifacts, and report validation slices from model evaluations and an anonymized human pilot. ForecastBench-Sim is intended to complement real-world forecasting benchmarks by providing controlled, immediately resolvable tasks for studying probabilistic reasoning under dynamic world states.

Related papers

• WorldSimProbe: Diagnosing Simulator Faithfulness in Action-Conditioned World Models for Embodied Manipulation [85.15806404962193]
  Action-conditioned world models (ACWMs) promise to provide embodied AI with scalable predictive simulators for planning, policy evaluation, and data generation.<n>We evaluate ACWMs through the observable capabilities expected of physical simulators.<n>WorldSimProbe reveals systematic action-realization degradation across control variation, structured failures in interaction grounding and dynamics.
  arXiv  Detail & Related papers  (2026-08-10T08:48:06Z)
• ReactSim-Bench: Benchmarking Reactive Behavior World Model Simulation in Autonomous Driving [56.96697737229895]
  Reactive capability is a key property of data-driven behavior world model simulators for autonomous driving simulation systems.<n>We introduce ReactSim-Bench for evaluating the reactive capability of behavior world model simulation in autonomous driving.
  arXiv  Detail & Related papers  (2026-06-12T03:11:06Z)
• World Models Meet Language Models: On the Complementarity of Concrete and Abstract Reasoning [67.93052893044603]
  We study how a model learns to invoke, verify, and integrate visual future simulation alongside abstract reasoning.<n>PF-OPSD uses ground-truth future videos and answers only as teacher-side privileged context to evaluate on-policy concrete-reasoning trajectories.
  arXiv  Detail & Related papers  (2026-06-02T13:07:49Z)
• RealBench: Benchmarking Data-Driven Numerical Weather Forecasting Under Operational Conditions and Extreme Event Challenges [31.389267895745252]
  We introduce RealBench, a next-generation benchmark for AI weather forecasting that emphasizes realistic evaluation under operational conditions.<n>RealBench features a strictly out-of-distribution test set spanning 2025 to eliminate data leakage and capture recent atmospheric regimes.<n>It integrates multiple data sources, including low-latency operational analysis and a large-scale global in-situ observation dataset comprising over 10,000 stations.
  arXiv  Detail & Related papers  (2026-05-24T08:46:17Z)
• FutureSim: Replaying World Events to Evaluate Adaptive Agents [69.52411849743358]
  We build FutureSim, where agents forecast world events beyond their knowledge cutoff while interacting with a chronological replay of the world.<n>FutureSim reveals a clear separation in their capabilities, with the best agent's accuracy being 25%, and many having worse Brier skill score than making no prediction at all.<n>Overall, we hope our benchmark design paves the way to measure AI progress on open-ended adaptation spanning long time-horizons in the real world.
  arXiv  Detail & Related papers  (2026-05-14T17:59:28Z)
• What if Tomorrow is the World Cup Final? Counterfactual Time Series Forecasting with Textual Conditions [17.093435375113298]
  Time series forecasting has become increasingly critical in real-world scenarios.<n>Traditional methods typically rely on historical data or factual future conditions.<n>We introduce the task of counterfactual time series forecasting with textual conditions.
  arXiv  Detail & Related papers  (2026-05-14T06:10:23Z)
• AGWM: Affordance-Grounded World Models for Environments with Compositional Prerequisites [18.635313325437664]
  We propose AGWM (Affordance-Grounded World Model), which learns an abstract affordance structure represented as a DAG of prerequisite dependencies to explicitly track the dynamic executability of actions.<n> Experiments on game-based simulated environments demonstrate the effectiveness of our method by achieving lower multi-step prediction error, better generalization to novel configurations, and improved interpretability.
  arXiv  Detail & Related papers  (2026-05-07T18:46:44Z)
• Towards Real-world Human Behavior Simulation: Benchmarking Large Language Models on Long-horizon, Cross-scenario, Heterogeneous Behavior Traces [81.41397370235102]
  We introduce OmniBehavior, the first user simulation benchmark constructed entirely from real-world data.<n>We show that current models struggle to accurately simulate complex behaviors, with performance plateauing even as context windows expand.<n>This results in the loss of individual differences and long-tail behaviors, highlighting critical directions for future high-fidelity simulation research.
  arXiv  Detail & Related papers  (2026-04-09T15:26:21Z)
• Impermanent: A Live Benchmark for Temporal Generalization in Time Series Forecasting [8.050703449557028]
  Impermanent is a live benchmark that evaluates forecasting models under open-world temporal change.<n>We focus on the top 400 repositories by star count and construct time series from issues opened, pull requests opened, push events, and new stargazers.<n>By shifting evaluation from static accuracy to sustained performance, Impermanent takes a concrete step toward assessing when and whether foundation-level generalization in time-series forecasting can be meaningfully claimed.
  arXiv  Detail & Related papers  (2026-03-09T17:59:00Z)
• TemporalBench: A Benchmark for Evaluating LLM-Based Agents on Contextual and Event-Informed Time Series Tasks [12.114998959919978]
  It is unclear whether strong forecasting performance reflects genuine temporal understanding or the ability to reason under contextual and event-driven conditions.<n>We introduce TemporalBench, a multi-domain benchmark designed to evaluate temporal reasoning behavior under progressively richer informational settings.<n>By controlling access to future targets and contextual information, the benchmark enables a diagnostic analysis of whether models can correctly interpret temporal patterns.
  arXiv  Detail & Related papers  (2026-02-05T01:02:19Z)
• GraphSCENE: On-Demand Critical Scenario Generation for Autonomous Vehicles in Simulation [11.896059467313668]
  This work introduces a novel method that generates dynamic temporal scene graphs corresponding to diverse traffic scenarios, on-demand, tailored to user-defined preferences.<n>A temporal Graph Neural Network (GNN) model learns to predict relationships between ego-vehicle agents and static structures, guided by real-world interaction patterns.<n>We render the predicted scenarios in simulation to further demonstrate their effectiveness as testing environments for AV agents.
  arXiv  Detail & Related papers  (2024-10-17T13:02:06Z)
• NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and Benchmarking [65.24988062003096]
  We present NAVSIM, a framework for benchmarking vision-based driving policies. Our simulation is non-reactive, i.e., the evaluated policy and environment do not influence each other. NAVSIM enabled a new competition held at CVPR 2024, where 143 teams submitted 463 entries, resulting in several new insights.
  arXiv  Detail & Related papers  (2024-06-21T17:59:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.