TFRBench: A Reasoning Benchmark for Evaluating Forecasting Systems

• URL: http://arxiv.org/abs/2604.05364v1
• Date: Tue, 07 Apr 2026 03:04:45 GMT
• Title: TFRBench: A Reasoning Benchmark for Evaluating Forecasting Systems
• Abstract summary: We introduce TFRBench, the first benchmark designed to evaluate the reasoning capabilities of forecasting systems.<n>We propose a systematic multi-agent framework that utilizes an iterative verification loop to synthesize numerically grounded reasoning traces.<n>Our evaluation confirms that this reasoning is causally effective; useful for evaluation; and prompting LLMs with our generated traces significantly improves forecasting accuracy.
• Score: 52.91956121737963
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce TFRBench, the first benchmark designed to evaluate the reasoning capabilities of forecasting systems. Traditionally, time-series forecasting has been evaluated solely on numerical accuracy, treating foundation models as ``black boxes.'' Unlike existing benchmarks, TFRBench provides a protocol for evaluating the reasoning generated by forecasting systems--specifically their analysis of cross-channel dependencies, trends, and external events. To enable this, we propose a systematic multi-agent framework that utilizes an iterative verification loop to synthesize numerically grounded reasoning traces. Spanning ten datasets across five domains, our evaluation confirms that this reasoning is causally effective; useful for evaluation; and prompting LLMs with our generated traces significantly improves forecasting accuracy compared to direct numerical prediction (e.g., avg. $\sim40.2\%\to56.6\%)$, validating the quality of our reasoning. Conversely, benchmarking experiments reveal that off-the-shelf LLMs consistently struggle with both reasoning (lower LLM-as-a-Judge scores) and numerical forecasting, frequently failing to capture domain-specific dynamics. TFRBench thus establishes a new standard for interpretable, reasoning-based evaluation in time-series forecasting. Our benchmark is available at: https://tfrbench.github.io

Related papers

• LLM-based Agents for Forecasting and Prediction: Methods, Training, Evaluation, and Applications [76.41731220830714]
  Large language models (LLMs) now support forecasting systems that combine language-based reasoning with temporal data, evidence retrieval, external tools, and iterative prediction.<n>We examine applications in finance, weather, health, energy, and operations, and we summarize the benchmarks and datasets used for evaluation.
  arXiv  Detail & Related papers  (2026-08-24T10:02:39Z)
• Playing Psychic: Using Thought Trees to Predict Reasoning Models Accuracy on Coding Tasks [7.675768582345005]
  We study frontier reasoning models to understand their performance on real-world coding benchmarks.<n>We propose structured thought-trees as means to represent reasoning traces.<n>We show that flagging and retrying structurally anomalous traces based on the extracted features yields consistent gains at lower complexity levels.
  arXiv  Detail & Related papers  (2026-04-18T09:30:36Z)
• Filtered Reasoning Score: Evaluating Reasoning Quality on a Model's Most-Confident Traces [14.997092856930061]
  Filtered Reasoning Score (FRS) computes reasoning quality using only the top-K% most confident traces.<n>We propose a reasoning score that evaluates reasoning traces along dimensions such as faithfulness, coherence, utility, and factuality.
  arXiv  Detail & Related papers  (2026-04-13T19:37:09Z)
• d-TreeRPO: Towards More Reliable Policy Optimization for Diffusion Language Models [45.27333046908981]
  emphd-TreeRPO is a reliable reinforcement learning framework for dLLMs.<n>We show that emphd-TreeRPO achieves significant gains on multiple reasoning benchmarks.
  arXiv  Detail & Related papers  (2025-12-10T14:20:07Z)
• Efficient Thought Space Exploration through Strategic Intervention [54.35208611253168]
  We propose a novel Hint-Practice Reasoning (HPR) framework that operationalizes this insight through two synergistic components.<n>The framework's core innovation lies in Distributional Inconsistency Reduction (DIR), which dynamically identifies intervention points.<n> Experiments across arithmetic and commonsense reasoning benchmarks demonstrate HPR's state-of-the-art efficiency-accuracy tradeoffs.
  arXiv  Detail & Related papers  (2025-11-13T07:26:01Z)
• Analyzing Uncertainty of LLM-as-a-Judge: Interval Evaluations with Conformal Prediction [13.958280616597385]
  This work presents the first framework to analyze the uncertainty by offering a prediction interval of LLM-based scoring via conformal prediction.<n>We perform extensive experiments and analysis, which show that conformal prediction can provide valid prediction interval with coverage guarantees.
  arXiv  Detail & Related papers  (2025-09-23T05:26:28Z)
• Aligning the Evaluation of Probabilistic Predictions with Downstream Value [2.6636053598505307]
  Metrics based solely on predictive performance often diverge from measures of real-world downstream impact.<n>We propose a data-driven method to learn a proxy evaluation function aligned with the downstream evaluation.<n>Our approach leverages weighted scoring rules parametrized by a neural network, where weighting is learned to align with the performance in the downstream task.
  arXiv  Detail & Related papers  (2025-08-25T17:41:27Z)
• Garbage In, Reasoning Out? Why Benchmark Scores are Unreliable and What to Do About It [1.6261897792391753]
  We conduct a systematic audit of three widely used reasoning benchmarks, SocialIQa, FauxPas-EAI, and ToMi.<n>We uncover pervasive flaws in both benchmark items and evaluation methodology.
  arXiv  Detail & Related papers  (2025-06-30T13:57:28Z)
• Learning to Plan & Reason for Evaluation with Thinking-LLM-as-a-Judge [78.28188747489769]
  We propose EvalPlanner, a preference optimization algorithm for Thinking-LLM-as-a-Judge.<n>In a self-training loop, EvalPlanner iteratively optimize over synthetically constructed evaluation plans and executions.<n>Our method achieves a new state-of-the-art performance for generative reward models on RewardBench.
  arXiv  Detail & Related papers  (2025-01-30T02:21:59Z)
• Consistency Checks for Language Model Forecasters [54.62507816753479]
  We measure the performance of forecasters in terms of the consistency of their predictions on different logically-related questions.<n>We build an automated evaluation system that generates a set of base questions, instantiates consistency checks from these questions, elicits predictions of the forecaster, and measures the consistency of the predictions.
  arXiv  Detail & Related papers  (2024-12-24T16:51:35Z)
• A Backdoor-based Explainable AI Benchmark for High Fidelity Evaluation of Attributions [60.06461883533697]
  We first identify a set of fidelity criteria that reliable benchmarks for attribution methods are expected to fulfill.<n>We then introduce a Backdoor-based eXplainable AI benchmark (BackX) that adheres to the desired fidelity criteria.<n>Our analysis also offers insights into defending against neural Trojans by utilizing the attributions.
  arXiv  Detail & Related papers  (2024-05-02T13:48:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.