Are We Winning the Wrong Game? Revisiting Evaluation Practices for Long-Term Time Series Forecasting

• URL: http://arxiv.org/abs/2603.08156v1
• Date: Mon, 09 Mar 2026 09:37:46 GMT
• Title: Are We Winning the Wrong Game? Revisiting Evaluation Practices for Long-Term Time Series Forecasting
• Abstract summary: Long-term time series forecasting (LTSF) is widely recognized as a central challenge in data mining and machine learning.<n>We argue that this metric-centric regime is not merely incomplete, but structurally misaligned with the broader objectives of forecasting.<n>We propose a multi-dimensional evaluation perspective that integrates statistical fidelity, structural coherence, and decision-level relevance.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Long-term time series forecasting (LTSF) is widely recognized as a central challenge in data mining and machine learning. LTSF has increasingly evolved into a benchmark-driven ''GAME,'' where models are ranked, compared, and declared state-of-the-art based primarily on marginal reductions in aggregated pointwise error metrics such as MSE and MAE. Across a small set of canonical datasets and fixed forecasting horizons, progress is communicated through leaderboard-style tables in which lower numerical scores define success. In this GAME, what is measured becomes what is optimized, and incremental error reduction becomes the dominant currency of advancement. We argue that this metric-centric regime is not merely incomplete, but structurally misaligned with the broader objectives of forecasting. In real-world settings, forecasting often prioritizes preserving temporal structure, trend stability, seasonal coherence, robustness to regime shifts, and supporting downstream decision processes. Optimizing aggregate pointwise error does not necessarily imply modeling these structural properties. As a result, leaderboard improvement may increasingly reflect specialization in benchmark configurations rather than a deeper understanding of temporal dynamics. This paper revisits LTSF evaluation as a foundational question in data science: what does it mean to measure forecasting progress? We propose a multi-dimensional evaluation perspective that integrates statistical fidelity, structural coherence, and decision-level relevance. By challenging the current metric monoculture, we aim to redirect attention from winning benchmark tables toward advancing meaningful, context-aware forecasting.

Related papers

• Orthogonal JEPA: Factorized Predictive States for Latent World Models [95.20158869896393]
  We introduce method, a latent world-modeling framework based on predictive factorization.<n>We show that method can allocate redundant capacity to dominant signals while providing weak or conflicting gradients to less dominant predictive structure.<n>Experiments on controlled vision, single-cell transcriptomics, longitudinal health records, continuous control, and molecular dynamics evaluate representation quality, forecasting, planning, and long-horizon stability.
  arXiv  Detail & Related papers  (2026-08-20T13:59:57Z)
• Improving Coherence in Hierarchical Time Series Forecasting using Structured Temporal Fusion [0.0]
  Hierarchical Temporal Fusion (HTF) integrates structured hierarchical embeddings with a coherence-aware loss function to ensure consistent forecasts across all levels of a hierarchy.<n>We evaluate HTF on two publicly available benchmark datasets: the M5 Walmart forecasting dataset and a publicly available hierarchical energy consumption dataset.
  arXiv  Detail & Related papers  (2026-06-26T19:11:40Z)
• Is Capability a Liability? More Capable Language Models Make Worse Forecasts When It Matters Most [11.4807141145402]
  In inverse scaling, more capable models produce worse distributional forecasts on forecasting problems with superlinear growth and tail risk of regime change.<n>The pattern appears on ForecastBench-Sim, in forecasting synthetic SIR epidemics with a matched linear control, and replicates in real-world datasets on COVID-19, measles, housing markets, and hyperinflation.<n>This inverse scaling does not appear on single-threshold metrics common in LLM forecasting benchmarks, reversing the sign of the capability--accuracy relationship on identical outputs.
  arXiv  Detail & Related papers  (2026-05-21T16:14:33Z)
• Nexus : An Agentic Framework for Time Series Forecasting [72.73790673303154]
  Time series forecasting requires reasoning with unstructured contextual data such as news or events.<n>We introduce Nexus, a multi-agent forecasting framework that decomposes prediction into specialized stages.<n>We show that current-generation LLMs possess substantially stronger intrinsic forecasting ability than previously recognized.
  arXiv  Detail & Related papers  (2026-05-14T05:12:13Z)
• LeapTS: Rethinking Time Series Forecasting as Adaptive Multi-Horizon Scheduling [74.94985663101906]
  We present LeapTS, a novel framework that reformulates time series forecasting as a dynamic scheduling process over the prediction horizon.<n>We show that LeapTS improves overall forecasting performance by at least 7.4% while achieving a 2.6$times$ to 5.3$times$ inference speedup over representative Transformer-based models.
  arXiv  Detail & Related papers  (2026-05-11T09:54:02Z)
• Evaluating Uplift Modeling under Structural Biases: Insights into Metric Stability and Model Robustness [8.135022024189306]
  In personalized marketing, uplift models estimate incremental effects by modeling how customer behavior changes under alternative treatments.<n>Real-world data often exhibit biases, such as selection bias, spillover effects, and unobserved confounding.
  arXiv  Detail & Related papers  (2026-03-21T11:54:25Z)
• Sequential Regression for Continuous Value Prediction using Residual Quantization [8.96389388600604]
  Continuous value prediction plays a crucial role in industrial-scale recommendation systems.<n>Existing generative approaches rely on rigid parametric distribution assumptions.<n>We propose a residual quantization (RQ)-based sequence learning framework that represents target continuous values as a sum of ordered quantization codes.
  arXiv  Detail & Related papers  (2026-02-26T13:52:54Z)
• Optimal Look-back Horizon for Time Series Forecasting in Federated Learning [26.070107882914844]
  This paper presents a principled framework for adaptive horizon selection in federated time series forecasting.<n>We derive a decomposition of the forecasting loss into a Bayesian term, which reflects irreducible uncertainty.<n>We prove that the total forecasting loss is minimized at the smallest horizon where the irreducible loss starts to saturate, while the approximation loss continues to rise.
  arXiv  Detail & Related papers  (2025-11-16T21:46:54Z)
• Beyond MSE: Ordinal Cross-Entropy for Probabilistic Time Series Forecasting [11.320830769077027]
  Current deep learning-based forecasting models primarily employ Mean Squared Error (MSE) loss functions for regression modeling.<n>We propose OCE-TS, a novel ordinal classification approach for time series forecasting that replaces MSE with Ordinal Cross-Entropy (OCE) loss.<n>Using MSE and Mean Absolute Error (MAE) as evaluation metrics, the results demonstrate that OCE-TS consistently outperforms benchmark models.
  arXiv  Detail & Related papers  (2025-11-13T11:14:24Z)
• ResAD: Normalized Residual Trajectory Modeling for End-to-End Autonomous Driving [64.42138266293202]
  ResAD is a Normalized Residual Trajectory Modeling framework.<n>It reframes the learning task to predict the residual deviation from an inertial reference.<n>On the NAVSIM benchmark, ResAD achieves a state-of-the-art PDMS of 88.6 using a vanilla diffusion policy.
  arXiv  Detail & Related papers  (2025-10-09T17:59:36Z)
• Accuracy Law for the Future of Deep Time Series Forecasting [65.46625911002202]
  Time series forecasting inherently faces a non-zero error lower bound due to its partially observable and uncertain nature.<n>This paper focuses on a fundamental question: how to estimate the performance upper bound of deep time series forecasting.<n>Based on rigorous statistical tests of over 2,800 newly trained deep forecasters, we discover a significant exponential relationship between the minimum forecasting error of deep models and the complexity of window-wise series patterns.
  arXiv  Detail & Related papers  (2025-10-03T05:18:47Z)
• Revisiting Multivariate Time Series Forecasting with Missing Values [65.30332997607141]
  Missing values are common in real-world time series.<n>Current approaches have developed an imputation-then-prediction framework that uses imputation modules to fill in missing values, followed by forecasting on the imputed data.<n>This framework overlooks a critical issue: there is no ground truth for the missing values, making the imputation process susceptible to errors that can degrade prediction accuracy.<n>We introduce Consistency-Regularized Information Bottleneck (CRIB), a novel framework built on the Information Bottleneck principle.
  arXiv  Detail & Related papers  (2025-09-27T20:57:48Z)
• Beyond Model Ranking: Predictability-Aligned Evaluation for Time Series Forecasting [18.018179328110048]
  We introduce a predictability-aligned diagnostic framework grounded in spectral coherence.<n>We provide the first systematic evidence of "predictability drift", demonstrating that a task's forecasting difficulty varies sharply over time.<n>Our evaluation reveals a key architectural trade-off: complex models are superior for low-predictability data, whereas linear models are highly effective on more predictable tasks.
  arXiv  Detail & Related papers  (2025-09-27T02:56:06Z)
• Improving Time Series Forecasting via Instance-aware Post-hoc Revision [44.90322487625981]
  Time series forecasting plays a vital role in various real-world applications.<n>Recent methods have achieved remarkable accuracy by incorporating advanced inductive biases and training strategies.<n>We propose a model-agnostic framework, PIR, designed to enhance forecasting performance through Post-forecasting Identification and Revision.
  arXiv  Detail & Related papers  (2025-05-29T15:56:41Z)
• Understanding, Predicting and Better Resolving Q-Value Divergence in Offline-RL [86.0987896274354]
  We first identify a fundamental pattern, self-excitation, as the primary cause of Q-value estimation divergence in offline RL. We then propose a novel Self-Excite Eigenvalue Measure (SEEM) metric to measure the evolving property of Q-network at training. For the first time, our theory can reliably decide whether the training will diverge at an early stage.
  arXiv  Detail & Related papers  (2023-10-06T17:57:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.