FutureSim: Replaying World Events to Evaluate Adaptive Agents

• URL: http://arxiv.org/abs/2605.15188v1
• Date: Thu, 14 May 2026 17:59:28 GMT
• Title: FutureSim: Replaying World Events to Evaluate Adaptive Agents
• Abstract summary: We build FutureSim, where agents forecast world events beyond their knowledge cutoff while interacting with a chronological replay of the world.<n>FutureSim reveals a clear separation in their capabilities, with the best agent's accuracy being 25%, and many having worse Brier skill score than making no prediction at all.<n>Overall, we hope our benchmark design paves the way to measure AI progress on open-ended adaptation spanning long time-horizons in the real world.
• Score: 69.52411849743358
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: AI agents are being increasingly deployed in dynamic, open-ended environments that require adapting to new information as it arrives. To efficiently measure this capability for realistic use-cases, we propose building grounded simulations that replay real-world events in the order they occurred. We build FutureSim, where agents forecast world events beyond their knowledge cutoff while interacting with a chronological replay of the world: real news articles arriving and questions resolving over the simulated period. We evaluate frontier agents in their native harness, testing their ability to predict world events over a three-month period from January to March 2026. FutureSim reveals a clear separation in their capabilities, with the best agent's accuracy being 25%, and many having worse Brier skill score than making no prediction at all. Through careful ablations, we show how FutureSim offers a realistic setting to study emerging research directions like long-horizon test-time adaptation, search, memory, and reasoning about uncertainty. Overall, we hope our benchmark design paves the way to measure AI progress on open-ended adaptation spanning long time-horizons in the real world.

Related papers

• ForecastBench-Sim: A Simulated-World Forecasting Benchmark [11.4807141145402]
  ForecastBench-Sim is a simulated-world forecasting benchmark built on game rollouts from Freeciv.<n>Forecasters receive a fixed world report and answer questions about hidden future states.<n>Because the world is simulated, the same setup can generate continuous or binary forecasting questions.
  arXiv  Detail & Related papers  (2026-06-17T04:52:41Z)
• FutureWorld: A Live Reinforcement Learning Environment for Predictive Agents with Real-World Outcome Rewards [20.541743597851177]
  We present FutureWorld, a live agentic reinforcement learning environment that closes the training loop between prediction, outcome realization, and parameter updates.<n>Across three open-source agents, successive FutureWorld training rounds lead to consistent improvements in prediction accuracy, probabilistic scoring, and calibration.
  arXiv  Detail & Related papers  (2026-04-29T14:34:45Z)
• World Reasoning Arena [36.28720055069025]
  WR-Arena is a benchmark for evaluating world models along three fundamental dimensions of next world simulation.<n>We build a task taxonomy and curate diverse datasets designed to probe these capabilities.<n>Our results expose a substantial gap between current models and human-level hypothetical reasoning.
  arXiv  Detail & Related papers  (2026-03-26T20:22:52Z)
• Current Agents Fail to Leverage World Model as Tool for Foresight [61.82522354207919]
  Generative world models offer a promising remedy: agents could use them to foresee outcomes before acting.<n>This paper empirically examines whether current agents can leverage such world models as tools to enhance their cognition.
  arXiv  Detail & Related papers  (2026-01-07T13:15:23Z)
• Dyna-Mind: Learning to Simulate from Experience for Better AI Agents [62.21219817256246]
  We argue that current AI agents need ''vicarious trial and error'' - the capacity to mentally simulate alternative futures before acting.<n>We introduce Dyna-Mind, a two-stage training framework that explicitly teaches (V)LM agents to integrate such simulation into their reasoning.
  arXiv  Detail & Related papers  (2025-10-10T17:30:18Z)
• Sim-to-Real Transfer of Deep Reinforcement Learning Agents for Online Coverage Path Planning [22.077058792635313]
  Coverage path planning is the problem of finding a path that covers the entire free space of a confined area.<n>We investigate the suitability of continuous-space reinforcement learning for this challenging problem.<n>We show that our approach surpasses the performance of both previous RL-based approaches and highly specialized methods.
  arXiv  Detail & Related papers  (2024-06-07T13:24:19Z)
• Predictive World Models from Real-World Partial Observations [66.80340484148931]
  We present a framework for learning a probabilistic predictive world model for real-world road environments. While prior methods require complete states as ground truth for learning, we present a novel sequential training method to allow HVAEs to learn to predict complete states from partially observed states only.
  arXiv  Detail & Related papers  (2023-01-12T02:07:26Z)
• Forecasting Future World Events with Neural Networks [68.43460909545063]
  Autocast is a dataset containing thousands of forecasting questions and an accompanying news corpus. The news corpus is organized by date, allowing us to precisely simulate the conditions under which humans made past forecasts. We test language models on our forecasting task and find that performance is far below a human expert baseline.
  arXiv  Detail & Related papers  (2022-06-30T17:59:14Z)
• TrafficSim: Learning to Simulate Realistic Multi-Agent Behaviors [74.67698916175614]
  We propose TrafficSim, a multi-agent behavior model for realistic traffic simulation. In particular, we leverage an implicit latent variable model to parameterize a joint actor policy. We show TrafficSim generates significantly more realistic and diverse traffic scenarios as compared to a diverse set of baselines.
  arXiv  Detail & Related papers  (2021-01-17T00:29:30Z)
• The Rise of AI-Driven Simulators: Building a New Crystal Ball [6.289422225292999]
  Continued U.S. and international prosperity, security, and health depend in part on continued improvements in simulation capabilities. The past decade has brought remarkable advances in complementary areas. In this paper, we lay out some themes that we envision forming part of a cohesive, multi-disciplinary, and application-inspired research agenda.
  arXiv  Detail & Related papers  (2020-12-11T00:13:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.