QuantSightBench: Evaluating LLM Quantitative Forecasting with Prediction Intervals

• URL: http://arxiv.org/abs/2604.15859v1
• Date: Fri, 17 Apr 2026 09:06:24 GMT
• Title: QuantSightBench: Evaluating LLM Quantitative Forecasting with Prediction Intervals
• Abstract summary: We propose prediction intervals as a more suitable evaluation format than point estimates for numerical forecasting.<n>To assess this capability, we introduce a new benchmark QuantSightBench, and evaluate frontier models under multiple settings.<n>None of the 11 evaluated frontier and open-weight models achieves the 90% coverage target, with the top performers Gemini 3.1 Pro (79.1%), Grok 4 (76.4%), and GPT-5.4 (75.3%) all falling at least 10 percentage points short.
• Score: 18.055205233907248
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Forecasting has become a natural benchmark for reasoning under uncertainty. Yet existing evaluations of large language models remain limited to judgmental tasks in simple formats, such as binary or multiple-choice questions. In practice, however, forecasting spans a far broader scope. Across domains such as economics, public health, and social demographics, decisions hinge on numerical estimates over continuous quantities, a capability that current benchmarks do not capture. Evaluating such estimates requires a format that makes uncertainty explicit and testable. We propose prediction intervals as a natural and rigorous interface for this purpose. They demand scale awareness, internal consistency across confidence levels, and calibration over a continuum of outcomes, making them a more suitable evaluation format than point estimates for numerical forecasting. To assess this capability, we introduce a new benchmark QuantSightBench, and evaluate frontier models under multiple settings, assessing both empirical coverage and interval sharpness. Our results show that none of the 11 evaluated frontier and open-weight models achieves the 90\% coverage target, with the top performers Gemini 3.1 Pro (79.1\%), Grok 4 (76.4\%), and GPT-5.4 (75.3\%) all falling at least 10 percentage points short. Calibration degrades sharply at extreme magnitudes, revealing systematic overconfidence across all evaluated models.

Related papers

• Multi-Dimensional Behavioral Evaluation of Agentic Stock Prediction Systems Using Large Language Model Judges with Closed-Loop Reinforcement Learning Feedback [1.2362187555287152]
  Forecast evaluation in finance has relied on aggregate accuracy metrics and predictive-accuracy tests built on point-forecast errors.<n>We propose a behavioral forecast-evaluation methodology that complements accuracy tests by assessing the intermediate decision process itself.
  arXiv  Detail & Related papers  (2026-05-07T06:31:34Z)
• Bi-Level Chaotic Fusion Based Graph Convolutional Network for Stock Market Prediction Interval [0.0]
  Financial market forecasting is inherently uncertain, yet most deep learning approaches rely on single estimates without uncertainty.<n>The problem can be solved using prediction intervals, which allow obtaining an uncertainty and lower bound for the prediction.<n>Yet, the current methods tend to disregard relationships between assets or cannot simultaneously ensure good calibration sharpness.
  arXiv  Detail & Related papers  (2026-05-05T05:06:42Z)
• TFRBench: A Reasoning Benchmark for Evaluating Forecasting Systems [52.91956121737963]
  We introduce TFRBench, the first benchmark designed to evaluate the reasoning capabilities of forecasting systems.<n>We propose a systematic multi-agent framework that utilizes an iterative verification loop to synthesize numerically grounded reasoning traces.<n>Our evaluation confirms that this reasoning is causally effective; useful for evaluation; and prompting LLMs with our generated traces significantly improves forecasting accuracy.
  arXiv  Detail & Related papers  (2026-04-07T03:04:45Z)
• Towards Integrating Uncertainty for Domain-Agnostic Segmentation [44.239195075597536]
  Foundation models for segmentation such as the Segment Anything Model (SAM) family exhibit strong zero-shot performance, but remain vulnerable in shifted or limited-knowledge domains.<n>This work investigates whether quantification uncertainty can mitigate such challenges and enhance model generalisability in a domain-agnostic manner.
  arXiv  Detail & Related papers  (2025-12-29T12:46:21Z)
• Localized Uncertainty Quantification in Random Forests via Proximities [1.0195618602298684]
  In machine learning, uncertainty quantification helps assess the reliability of model predictions.<n>Traditional approaches often emphasize predictive accuracy, but there is a growing focus on incorporating uncertainty measures.<n>We propose a new approach using naturally occurring test sets and similarity measures (proximities) typically viewed as byproducts of random forests.
  arXiv  Detail & Related papers  (2025-09-26T20:53:28Z)
• FOReCAst: The Future Outcome Reasoning and Confidence Assessment Benchmark [11.149409619312827]
  FOReCAst is a benchmark that evaluates models' ability to make predictions and their confidence in them.<n>It spans diverse forecasting scenarios involving Boolean questions, timeframe prediction, and quantity estimation.<n>It provides a comprehensive evaluation of both prediction accuracy and confidence calibration for real-world applications.
  arXiv  Detail & Related papers  (2025-02-27T01:36:00Z)
• Score Matching-based Pseudolikelihood Estimation of Neural Marked Spatio-Temporal Point Process with Uncertainty Quantification [59.81904428056924]
  We introduce SMASH: a Score MAtching estimator for learning markedPs with uncertainty quantification. Specifically, our framework adopts a normalization-free objective by estimating the pseudolikelihood of markedPs through score-matching. The superior performance of our proposed framework is demonstrated through extensive experiments in both event prediction and uncertainty quantification.
  arXiv  Detail & Related papers  (2023-10-25T02:37:51Z)
• Toward Reliable Human Pose Forecasting with Uncertainty [51.628234388046195]
  We develop an open-source library for human pose forecasting, including multiple models, supporting several datasets. We devise two types of uncertainty in the problem to increase performance and convey better trust.
  arXiv  Detail & Related papers  (2023-04-13T17:56:08Z)
• The Implicit Delta Method [61.36121543728134]
  In this paper, we propose an alternative, the implicit delta method, which works by infinitesimally regularizing the training loss of uncertainty. We show that the change in the evaluation due to regularization is consistent for the variance of the evaluation estimator, even when the infinitesimal change is approximated by a finite difference.
  arXiv  Detail & Related papers  (2022-11-11T19:34:17Z)
• Dense Uncertainty Estimation [62.23555922631451]
  In this paper, we investigate neural networks and uncertainty estimation techniques to achieve both accurate deterministic prediction and reliable uncertainty estimation. We work on two types of uncertainty estimations solutions, namely ensemble based methods and generative model based methods, and explain their pros and cons while using them in fully/semi/weakly-supervised framework.
  arXiv  Detail & Related papers  (2021-10-13T01:23:48Z)
• Quantifying Uncertainty in Deep Spatiotemporal Forecasting [67.77102283276409]
  We describe two types of forecasting problems: regular grid-based and graph-based. We analyze UQ methods from both the Bayesian and the frequentist point view, casting in a unified framework via statistical decision theory. Through extensive experiments on real-world road network traffic, epidemics, and air quality forecasting tasks, we reveal the statistical computational trade-offs for different UQ methods.
  arXiv  Detail & Related papers  (2021-05-25T14:35:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.