Expectations vs. Realities: The Cost of MSE-Optimal Forecasting Under Conditional Uncertainty

• URL: http://arxiv.org/abs/2606.04342v1
• Date: Wed, 03 Jun 2026 01:50:32 GMT
• Title: Expectations vs. Realities: The Cost of MSE-Optimal Forecasting Under Conditional Uncertainty
• Abstract summary: Multi-step time series forecasting (MSF) is commonly evaluated using pointwise error metrics such as mean squared error (MSE)<n>We show that this can be misleading under conditional uncertainty, where the conditional expectation becomes unrepresentative of typical realized values at longer horizons.<n>This establishes a fundamental, model-agnostic trade-off between point accuracy and marginal realism in MSF evaluation.
• Score: 1.2489632787815885
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multi-step time series forecasting (MSF) is commonly evaluated using point-wise error metrics such as mean squared error (MSE), implicitly treating the conditional mean as a sufficient target. We show that this can be misleading under conditional uncertainty, where the conditional expectation becomes unrepresentative of typical realized values at longer horizons. We formalize this effect through a conditional uncertainty gap and prove that whenever this gap is nonzero, no deterministic predictor can simultaneously minimize MSE and match the marginal distribution of realized futures. This establishes a fundamental, model-agnostic trade-off between point accuracy and marginal realism in MSF evaluation. Using controlled stochastic dynamical systems and nine real-world forecasting benchmarks, we empirically characterize the resulting accuracy--realism frontier and \textbf{quantify the practical cost of MSE-only model selection}. As conditional uncertainty increases with forecast horizon, the attainable set expands into a pronounced Pareto front, separating MSE-optimal but under-dispersed predictors from methods that trade accuracy for realistic marginal variability. \textbf{Across benchmarks, we find that small relaxations in MSE ($\boldsymbol{\le 5\%}$) frequently unlock disproportionate gains in marginal realism, with median improvements of $\mathbf{17.3\%}$ and gains exceeding $\mathbf{30\%}$ in some datasets.} We further show that common forecasting strategies systematically occupy different regions of this frontier: direct multi-output predictors concentrate near the accuracy-optimal extreme, while recursive strategies and sample-based inference favors marginal realism. Together, these results expose a structural failure mode of MSE-based evaluation in long-horizon forecasting and recast strategy and inference selection as navigation of an unavoidable accuracy--realism trade-off.

Related papers

• Beyond MSE: Rethinking the Evaluation Metric and Benchmarking for Irregular Time Series Forecasting [8.684367575006542]
  Existing benchmarks typically use mean squared error (MSE) as the evaluation metric.<n>We show that MSE is determined not only by the model prediction but also by the sample-specific sampling distributions.<n>To address this issue, we propose the Continuous-time Squared Error (CSE)<n> Experiments show that CSE can recover continuous-time risk more accurately than MSE.
  arXiv  Detail & Related papers  (2026-08-18T02:47:04Z)
• Two-stage Odd Residual Flows for Mean-Preserving Probabilistic Time Series Forecasting [9.992963275755159]
  Two-stage Odd Residual Flows (TORF) is a framework that decouples mean forecasting from uncertainty estimation.<n>TORF achieves state-of-the-art deterministic accuracy (NMAE) while providing strong density estimation performance (CRPS) on short and long-horizon forecasting.
  arXiv  Detail & Related papers  (2026-08-11T16:22:47Z)
• Quantifying the Risk-Return Tradeoff in Forecasting [0.0]
  Average forecast accuracy is not the same as forecast reliability.<n>I treat forecast loss differentials relative to a benchmark as a return series.<n>I then evaluate these returns using risk-adjusted performance measures from finance, including the Sharpe ratio, Sortino ratio, Omega ratio, and drawdown-based metrics.
  arXiv  Detail & Related papers  (2026-05-10T19:21:12Z)
• The Illusion of Certainty: Uncertainty quantification for LLMs fails under ambiguity [48.899855816199484]
  We introduce MAQA* and AmbigQA*, the first ambiguous question-answering (QA) datasets equipped with ground-truth answer distributions.<n>We show that predictive-distribution and ensemble-based estimators are fundamentally limited under ambiguity.
  arXiv  Detail & Related papers  (2025-11-06T14:46:35Z)
• RDIT: Residual-based Diffusion Implicit Models for Probabilistic Time Series Forecasting [4.140149411004857]
  RDIT is a plug-and-play framework that combines point estimation and residual-based conditional diffusion with a bidirectional Mamba network.<n>We show that RDIT achieves lower CRPS, rapid inference, and improved coverage compared to strong baselines.
  arXiv  Detail & Related papers  (2025-09-02T14:06:29Z)
• Probabilistic Scores of Classifiers, Calibration is not Enough [0.32985979395737786]
  In binary classification tasks, accurate representation of probabilistic predictions is essential for various real-world applications. In this study, we highlight approaches that prioritize the alignment between predicted scores and true probability distributions. Our findings reveal limitations in traditional calibration metrics, which could undermine the reliability of predictive models for critical decision-making.
  arXiv  Detail & Related papers  (2024-08-06T19:53:00Z)
• Probabilistic Conformal Prediction with Approximate Conditional Validity [81.30551968980143]
  We develop a new method for generating prediction sets that combines the flexibility of conformal methods with an estimate of the conditional distribution. Our method consistently outperforms existing approaches in terms of conditional coverage.
  arXiv  Detail & Related papers  (2024-07-01T20:44:48Z)
• SEMF: Supervised Expectation-Maximization Framework for Predicting Intervals [0.254890465057467]
  Supervised Expectation-Maximization Framework (SEMF) is a versatile and model-agnostic approach for generating prediction intervals with any ML model.<n>SEMF consistently produces narrower prediction intervals while maintaining the desired coverage probability.<n>Without using the quantile (pinball) loss, SEMF allows point predictors, including gradient-boosted trees and neural networks, to be calibrated with conformal quantile regression.
  arXiv  Detail & Related papers  (2024-05-28T13:43:34Z)
• When Does Confidence-Based Cascade Deferral Suffice? [69.28314307469381]
  Cascades are a classical strategy to enable inference cost to vary adaptively across samples. A deferral rule determines whether to invoke the next classifier in the sequence, or to terminate prediction. Despite being oblivious to the structure of the cascade, confidence-based deferral often works remarkably well in practice.
  arXiv  Detail & Related papers  (2023-07-06T04:13:57Z)
• On the Variance, Admissibility, and Stability of Empirical Risk Minimization [57.63331017830154]
  Empirical Risk Minimization (ERM) may attain minimax suboptimal rates in terms of the mean squared error.<n>We prove that under relatively mild assumptions, the suboptimality of ERM must be due to its large bias.
  arXiv  Detail & Related papers  (2023-05-29T15:25:48Z)
• Probable Domain Generalization via Quantile Risk Minimization [90.15831047587302]
  Domain generalization seeks predictors which perform well on unseen test distributions. We propose a new probabilistic framework for DG where the goal is to learn predictors that perform well with high probability.
  arXiv  Detail & Related papers  (2022-07-20T14:41:09Z)
• Uncertainty estimation of pedestrian future trajectory using Bayesian approximation [137.00426219455116]
  Under dynamic traffic scenarios, planning based on deterministic predictions is not trustworthy. The authors propose to quantify uncertainty during forecasting using approximation which deterministic approaches fail to capture. The effect of dropout weights and long-term prediction on future state uncertainty has been studied.
  arXiv  Detail & Related papers  (2022-05-04T04:23:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.