Why Model Selection Fails in Time Series Forecasting: An Empirical Study of Instability Across Data Regimes

• URL: http://arxiv.org/abs/2605.01608v1
• Date: Sat, 02 May 2026 21:19:29 GMT
• Title: Why Model Selection Fails in Time Series Forecasting: An Empirical Study of Instability Across Data Regimes
• Abstract summary: Time series forecasting models often exhibit inconsistent performance across datasets with varying statistical and structural properties.<n>This paper investigates why rule-based model selection fails in time series forecasting by analyzing the relationship between data-regime descriptors and model performance.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Time series forecasting models often exhibit inconsistent performance across datasets with varying statistical and structural properties. Despite the wide range of available forecasting techniques, it remains unclear whether model selection can be reliably guided by simple data characteristics. This paper investigates why rule-based model selection fails in time series forecasting by analyzing the relationship between data-regime descriptors and model performance. A descriptor-based framework is introduced to characterize time series using measurable properties, including trend strength, seasonality, noise level, and temporal dependence. Based on these descriptors, a rule-based selection mechanism is formulated to map data regimes to candidate forecasting models. The approach is evaluated on multiple real-world datasets across different domains and forecasting horizons. The results show that rule-based model selection achieves low accuracy, with correct model identification occurring in only a small fraction of cases. Significant discrepancies are observed between recommended and empirically optimal models, particularly in noisy and mixed regimes. Further analysis reveals that model performance is highly sensitive to both dataset characteristics and forecasting horizon, resulting in substantial ranking instability across scenarios. These findings explain why simple heuristic rules fail to generalize and demonstrate that forecasting performance cannot be reliably predicted using static, descriptor-based approaches. This study provides empirical evidence that model selection in time series forecasting is inherently context-dependent and highlights the need for more adaptive, data-driven strategies.

Related papers

• Data Leakage Inflates Generalizability of Power Outage Prediction Models [0.9558392439655014]
  We identify three common methodological choices in power outage prediction models that influence their ability to generalize.<n>We compare the predictive performance impacts of different methodological decisions using publicly available data for the U.S. East Coast from 2018 to 2023.<n>While random train-test splits yield strong performance, we show that these results are inflated by spatial and temporal autocorrelation.<n>These findings suggest that, given current data availability and evaluation practices, publicly trained outage prediction models offer limited and uncertain operational value.
  arXiv  Detail & Related papers  (2026-08-25T15:05:55Z)
• Model Correlation Detection via Random Selection Probing [62.093777777813756]
  Existing similarity-based methods require access to model parameters or produce scores without thresholds.<n>We introduce Random Selection Probing (RSP), a hypothesis-testing framework that formulates model correlation detection as a statistical test.<n>RSP produces rigorous p-values that quantify evidence of correlation.
  arXiv  Detail & Related papers  (2025-09-29T01:40:26Z)
• Estimating Time Series Foundation Model Transferability via In-Context Learning [74.65355820906355]
  Time series foundation models (TSFMs) offer strong zero-shot forecasting via large-scale pre-training.<n>Fine-tuning remains critical for boosting performance in domains with limited public data.<n>We introduce TimeTic, a transferability estimation framework that recasts model selection as an in-context-learning problem.
  arXiv  Detail & Related papers  (2025-09-28T07:07:13Z)
• ARIES: Relation Assessment and Model Recommendation for Deep Time Series Forecasting [54.57031153712623]
  ARIES is a framework for assessing relation between time series properties and modeling strategies.<n>We propose the first deep forecasting model recommender, capable of providing interpretable suggestions for real-world time series.
  arXiv  Detail & Related papers  (2025-09-07T13:57:14Z)
• ModelRadar: Aspect-based Forecast Evaluation [0.393259574660092]
  Current practices for evaluating and comparing forecasting models focus on summarising performance into a single score.<n>This limitation is especially problematic for time series forecasting, where multiple layers of averaging--across time steps, horizons, and multiple time series in a dataset--can mask relevant performance variations.<n>We propose ModelRadar, a framework for evaluating univariate time series forecasting models across multiple aspects, such as stationarity, presence of anomalies, or forecasting horizons.
  arXiv  Detail & Related papers  (2025-03-31T11:50:45Z)
• Deep Non-Parametric Time Series Forecaster [19.800783133682955]
  The proposed approach does not assume any parametric form for the predictive distribution and instead generates predictions by sampling from the empirical distribution according to a tunable strategy. We develop a global version of the proposed method that automatically learns the sampling strategy by exploiting the information across multiple related time series.
  arXiv  Detail & Related papers  (2023-12-22T12:46:30Z)
• Towards a performance characteristic curve for model evaluation: an application in information diffusion prediction [2.8686437689115354]
  We propose a metric based on information entropy to quantify the randomness in diffusion data.<n>We then identify a scaling pattern between the randomness and the prediction accuracy of the model.<n>The validity of the curve is tested by three prediction models in the same family.
  arXiv  Detail & Related papers  (2023-09-18T07:32:57Z)
• Assessment of Spatio-Temporal Predictors in the Presence of Missing and Heterogeneous Data [23.280400290071732]
  Deep learning approaches achieve outstanding predictive performance in modeling modern data, despite increasing complexity and scale.<n> evaluating the quality of predictive models becomes more challenging as traditional statistical assumptions often no longer hold.<n>This paper introduces a residual analysis framework designed to assess the optimality of temporal-temporal predictive neural models.
  arXiv  Detail & Related papers  (2023-02-03T12:55:08Z)
• TACTiS: Transformer-Attentional Copulas for Time Series [76.71406465526454]
  estimation of time-varying quantities is a fundamental component of decision making in fields such as healthcare and finance. We propose a versatile method that estimates joint distributions using an attention-based decoder. We show that our model produces state-of-the-art predictions on several real-world datasets.
  arXiv  Detail & Related papers  (2022-02-07T21:37:29Z)
• Selecting the suitable resampling strategy for imbalanced data classification regarding dataset properties [62.997667081978825]
  In many application domains such as medicine, information retrieval, cybersecurity, social media, etc., datasets used for inducing classification models often have an unequal distribution of the instances of each class. This situation, known as imbalanced data classification, causes low predictive performance for the minority class examples. Oversampling and undersampling techniques are well-known strategies to deal with this problem by balancing the number of examples of each class.
  arXiv  Detail & Related papers  (2021-12-15T18:56:39Z)
• Simultaneously Reconciled Quantile Forecasting of Hierarchically Related Time Series [11.004159006784977]
  We propose a flexible nonlinear model that optimize quantile regression loss coupled with suitable regularization terms to maintain consistency of forecasts across hierarchies. The theoretical framework introduced herein can be applied to any forecasting model with an underlying differentiable loss function.
  arXiv  Detail & Related papers  (2021-02-25T00:59:01Z)
• Characterizing Fairness Over the Set of Good Models Under Selective Labels [69.64662540443162]
  We develop a framework for characterizing predictive fairness properties over the set of models that deliver similar overall performance. We provide tractable algorithms to compute the range of attainable group-level predictive disparities. We extend our framework to address the empirically relevant challenge of selectively labelled data.
  arXiv  Detail & Related papers  (2021-01-02T02:11:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.