Dr-CiK: A Testbed for Foresight-Driven Agents

• URL: http://arxiv.org/abs/2605.27904v1
• Date: Wed, 27 May 2026 03:26:42 GMT
• Title: Dr-CiK: A Testbed for Foresight-Driven Agents
• Abstract summary: We introduce Dr-CiK, a benchmark for evaluating whether agents can retrieve forecasting-relevant context from a document corpus.<n>We show that high-quality context substantially improves forecasting performance in Dr-CiK.<n>Our results motivate research on foresight-driven agents that search for the right context to predict the future.
• Score: 58.303939183596015
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Time series forecasting in real-world settings often depends not only on historical observations, but also on external context that must be actively discovered from noisy, heterogeneous information sources. Yet existing context-aided forecasting benchmarks typically assume that the supporting context is already provided, leaving open whether agents can identify it on their own. Therefore, we introduce Dr-CiK, a benchmark for evaluating whether agents can retrieve forecasting-relevant supporting context from a document corpus, filter out distractors, distill the retrieved context into forecast-useful evidence, and generate forecasts supported by that evidence. Through context ablations and evaluations of state-of-the-art deep research and forecasting methods paired together, we show that high-quality context substantially improves forecasting performance in Dr-CiK. However, most existing DR agents recover only a small fraction of the ground-truth supporting evidence (usually <5%), are frequently misled by distractors (>80% distractor citations), and can cause forecasters to perform worse with retrieved context than without context. Our results motivate research on foresight-driven agents that search for the right context to predict the future.

Related papers

• From Narrative to Auditable Forecasts: A Structured Scaffold for Agentic Forecasting [28.10023756949099]
  AuditForecast is an agentic scaffold for structured probabilistic forecasting.<n>It anchors the forecast with a suitable quantitative baseline model, uses model-guided data retrieval to derive a base probability, and then applies situational factor updates outside the model's scope.<n> AuditForecast produces an auditable forecasting report that makes forecast construction explicit and supports systematic post hoc analysis.
  arXiv  Detail & Related papers  (2026-09-05T05:58:39Z)
• FRESCO: Benchmarking and Optimizing Re-rankers for Evolving Semantic Conflict in Retrieval-Augmented Generation [73.22935457705057]
  We introduce FRESCO, a benchmark for evaluating re-rankers in temporally dynamic contexts.<n>By pairing recency-seeking queries with historical Wikipedia revisions, FRESCO tests whether re-rankers can prioritize factually recent evidence while maintaining semantic relevance.<n>Our evaluation reveals a consistent failure mode across existing re-rankers: a strong bias toward older, semantically rich documents, even when they are factually obsolete.
  arXiv  Detail & Related papers  (2026-04-14T17:04:25Z)
• TemporalBench: A Benchmark for Evaluating LLM-Based Agents on Contextual and Event-Informed Time Series Tasks [12.114998959919978]
  It is unclear whether strong forecasting performance reflects genuine temporal understanding or the ability to reason under contextual and event-driven conditions.<n>We introduce TemporalBench, a multi-domain benchmark designed to evaluate temporal reasoning behavior under progressively richer informational settings.<n>By controlling access to future targets and contextual information, the benchmark enables a diagnostic analysis of whether models can correctly interpret temporal patterns.
  arXiv  Detail & Related papers  (2026-02-05T01:02:19Z)
• Proof of Time: A Benchmark for Evaluating Scientific Idea Judgments [12.46263502996029]
  PoT is a semi-verifiable benchmarking framework for large language models.<n>It links scientific idea judgments to downstream signals that become observable later.<n>PoT supports scalable evaluation of agents on future-facing scientific idea judgment tasks.
  arXiv  Detail & Related papers  (2026-01-12T14:55:37Z)
• PROPHET: An Inferable Future Forecasting Benchmark with Causal Intervened Likelihood Estimation [46.3251656496956]
  Recent advances in large language model (LLM)-based systems have shown remarkable potential in forecasting future events.<n>Several benchmarks have been established to evaluate the forecasting capabilities by formalizing the event prediction as a retrieval-augmented generation (RAG) and reasoning task.<n>We introduce a new benchmark, PROPHET, which comprises inferable forecasting questions paired with relevant news for retrieval.
  arXiv  Detail & Related papers  (2025-04-02T08:57:42Z)
• Detecting Multimodal Situations with Insufficient Context and Abstaining from Baseless Predictions [75.45274978665684]
  Vision-Language Understanding (VLU) benchmarks contain samples where answers rely on assumptions unsupported by the provided context.<n>We collect contextual data for each sample whenever available and train a context selection module to facilitate evidence-based model predictions.<n>We develop a general-purpose Context-AwaRe Abstention detector to identify samples lacking sufficient context and enhance model accuracy.
  arXiv  Detail & Related papers  (2024-05-18T02:21:32Z)
• Streaming Motion Forecasting for Autonomous Driving [71.7468645504988]
  We introduce a benchmark that queries future trajectories on streaming data and we refer to it as "streaming forecasting" Our benchmark inherently captures the disappearance and re-appearance of agents, which is a safety-critical problem yet overlooked by snapshot-based benchmarks. We propose a plug-and-play meta-algorithm called "Predictive Streamer" that can adapt any snapshot-based forecaster into a streaming forecaster.
  arXiv  Detail & Related papers  (2023-10-02T17:13:16Z)
• A Closer Look at Debiased Temporal Sentence Grounding in Videos: Dataset, Metric, and Approach [53.727460222955266]
  Temporal Sentence Grounding in Videos (TSGV) aims to ground a natural language sentence in an untrimmed video. Recent studies have found that current benchmark datasets may have obvious moment annotation biases. We introduce a new evaluation metric "dR@n,IoU@m" that discounts the basic recall scores to alleviate the inflating evaluation caused by biased datasets.
  arXiv  Detail & Related papers  (2022-03-10T08:58:18Z)
• SAIS: Supervising and Augmenting Intermediate Steps for Document-Level Relation Extraction [51.27558374091491]
  We propose to explicitly teach the model to capture relevant contexts and entity types by supervising and augmenting intermediate steps (SAIS) for relation extraction. Based on a broad spectrum of carefully designed tasks, our proposed SAIS method not only extracts relations of better quality due to more effective supervision, but also retrieves the corresponding supporting evidence more accurately.
  arXiv  Detail & Related papers  (2021-09-24T17:37:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.