Beyond MSE: Rethinking the Evaluation Metric and Benchmarking for Irregular Time Series Forecasting

• URL: http://arxiv.org/abs/2608.17293v1
• Date: Tue, 18 Aug 2026 02:47:04 GMT
• Title: Beyond MSE: Rethinking the Evaluation Metric and Benchmarking for Irregular Time Series Forecasting
• Abstract summary: Existing benchmarks typically use mean squared error (MSE) as the evaluation metric.<n>We show that MSE is determined not only by the model prediction but also by the sample-specific sampling distributions.<n>To address this issue, we propose the Continuous-time Squared Error (CSE)<n> Experiments show that CSE can recover continuous-time risk more accurately than MSE.
• Score: 8.684367575006542
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Existing research on irregular time-series forecasting has primarily focused on model design, while evaluation metrics remain insufficiently studied. Existing benchmarks typically use mean squared error (MSE) as the evaluation metric. We show that, in irregular forecasting, MSE is determined not only by the model prediction but also by the sample-specific timestamp sampling distributions, leading to a biased assessment of the models' continuous-time predictive performance. To address this issue, we propose the Continuous-time Squared Error (CSE), which employs importance weighting to eliminate the influence of the timestamp sampling distributions. We further theoretically prove that CSE's asymptotic estimation error with respect to continuous-time risk is no greater than that of MSE. Finally, we construct a systematic benchmark covering synthetic, semi-synthetic, and eight real-world datasets to validate the effectiveness of CSE and systematically evaluate models' continuous-time predictive performance. Experiments show that CSE can recover continuous-time risk more accurately than MSE, while relying solely on MSE may not fully reflect models' continuous-time predictive performance in real-world scenarios. Our code can be obtained at https://github.com/hnu-vis/ITS-Bench.

Related papers

• Latent-Regime Bias Auditing for Volatility Forecasting [0.44333156307871585]
  This paper proposes a model-agnostic audit framework for evaluating whether volatility forecasts remain reliable across latent market regimes.<n>We learn time-series representations of market-state windows, cluster them into regimes using only training information, and compare aggregate forecast behavior with regime-conditional bias, tail-underprediction, and underprediction-sensitive economic losses.
  arXiv  Detail & Related papers  (2026-08-03T02:09:39Z)
• Expectations vs. Realities: The Cost of MSE-Optimal Forecasting Under Conditional Uncertainty [1.2489632787815885]
  Multi-step time series forecasting (MSF) is commonly evaluated using pointwise error metrics such as mean squared error (MSE)<n>We show that this can be misleading under conditional uncertainty, where the conditional expectation becomes unrepresentative of typical realized values at longer horizons.<n>This establishes a fundamental, model-agnostic trade-off between point accuracy and marginal realism in MSF evaluation.
  arXiv  Detail & Related papers  (2026-06-03T01:50:32Z)
• LEAF: A Living Benchmark for Event-Augmented Forecasting [78.94019460534442]
  LEAF is the first living benchmark for event-augmented forecasting tasks.<n>We evaluate state-of-the-art proprietary and open-weight LLMs.<n>We find that LLMs achieve better performance on equities they confidently identify as more predictable.
  arXiv  Detail & Related papers  (2026-05-09T03:17:59Z)
• Beyond MSE: Ordinal Cross-Entropy for Probabilistic Time Series Forecasting [11.320830769077027]
  Current deep learning-based forecasting models primarily employ Mean Squared Error (MSE) loss functions for regression modeling.<n>We propose OCE-TS, a novel ordinal classification approach for time series forecasting that replaces MSE with Ordinal Cross-Entropy (OCE) loss.<n>Using MSE and Mean Absolute Error (MAE) as evaluation metrics, the results demonstrate that OCE-TS consistently outperforms benchmark models.
  arXiv  Detail & Related papers  (2025-11-13T11:14:24Z)
• Time Series Foundation Models: Benchmarking Challenges and Requirements [0.0]
  Time Series Foundation Models (TSFMs) represent a new paradigm for time series forecasting.<n> evaluating TSFMs is tricky, as with ever more extensive training sets, it becomes more challenging to ensure integrity benchmarking data.
  arXiv  Detail & Related papers  (2025-10-15T15:15:45Z)
• Revisiting Multivariate Time Series Forecasting with Missing Values [65.30332997607141]
  Missing values are common in real-world time series.<n>Current approaches have developed an imputation-then-prediction framework that uses imputation modules to fill in missing values, followed by forecasting on the imputed data.<n>This framework overlooks a critical issue: there is no ground truth for the missing values, making the imputation process susceptible to errors that can degrade prediction accuracy.<n>We introduce Consistency-Regularized Information Bottleneck (CRIB), a novel framework built on the Information Bottleneck principle.
  arXiv  Detail & Related papers  (2025-09-27T20:57:48Z)
• Evidential time-to-event prediction with calibrated uncertainty quantification [12.446406577462069]
  Time-to-event analysis provides insights into clinical prognosis and treatment recommendations.<n>We propose an evidential regression model specifically designed for time-to-event prediction.<n>We show that our model delivers both accurate and reliable performance, outperforming state-of-the-art methods.
  arXiv  Detail & Related papers  (2024-11-12T15:06:04Z)
• Score Matching-based Pseudolikelihood Estimation of Neural Marked Spatio-Temporal Point Process with Uncertainty Quantification [59.81904428056924]
  We introduce SMASH: a Score MAtching estimator for learning markedPs with uncertainty quantification. Specifically, our framework adopts a normalization-free objective by estimating the pseudolikelihood of markedPs through score-matching. The superior performance of our proposed framework is demonstrated through extensive experiments in both event prediction and uncertainty quantification.
  arXiv  Detail & Related papers  (2023-10-25T02:37:51Z)
• The Implicit Delta Method [61.36121543728134]
  In this paper, we propose an alternative, the implicit delta method, which works by infinitesimally regularizing the training loss of uncertainty. We show that the change in the evaluation due to regularization is consistent for the variance of the evaluation estimator, even when the infinitesimal change is approximated by a finite difference.
  arXiv  Detail & Related papers  (2022-11-11T19:34:17Z)
• Learning Interpretable Deep State Space Model for Probabilistic Time Series Forecasting [98.57851612518758]
  Probabilistic time series forecasting involves estimating the distribution of future based on its history. We propose a deep state space model for probabilistic time series forecasting whereby the non-linear emission model and transition model are parameterized by networks. We show in experiments that our model produces accurate and sharp probabilistic forecasts.
  arXiv  Detail & Related papers  (2021-01-31T06:49:33Z)
• A Worrying Analysis of Probabilistic Time-series Models for Sales Forecasting [10.690379201437015]
  Probabilistic time-series models become popular in the forecasting field as they help to make optimal decisions under uncertainty. We analyze the performance of three prominent probabilistic time-series models for sales forecasting.
  arXiv  Detail & Related papers  (2020-11-21T03:31:23Z)
• Performance metrics for intervention-triggering prediction models do not reflect an expected reduction in outcomes from using the model [71.9860741092209]
  Clinical researchers often select among and evaluate risk prediction models. Standard metrics calculated from retrospective data are only related to model utility under certain assumptions. When predictions are delivered repeatedly throughout time, the relationship between standard metrics and utility is further complicated.
  arXiv  Detail & Related papers  (2020-06-02T16:26:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.