Forecasting with Deep Learning: Beyond Average of Average of Average Performance

• URL: http://arxiv.org/abs/2406.16590v1
• Date: Mon, 24 Jun 2024 12:28:22 GMT
• Title: Forecasting with Deep Learning: Beyond Average of Average of Average Performance
• Authors: Vitor Cerqueira, Luis Roque, Carlos Soares,
• Abstract summary: Current practices for evaluating and comparing forecasting models focus on summarising performance into a single score. We propose a novel framework for evaluating models from multiple perspectives. We show the advantages of this framework by comparing a state-of-the-art deep learning approach with classical forecasting techniques.
• Score: 0.393259574660092
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Accurate evaluation of forecasting models is essential for ensuring reliable predictions. Current practices for evaluating and comparing forecasting models focus on summarising performance into a single score, using metrics such as SMAPE. We hypothesize that averaging performance over all samples dilutes relevant information about the relative performance of models. Particularly, conditions in which this relative performance is different than the overall accuracy. We address this limitation by proposing a novel framework for evaluating univariate time series forecasting models from multiple perspectives, such as one-step ahead forecasting versus multi-step ahead forecasting. We show the advantages of this framework by comparing a state-of-the-art deep learning approach with classical forecasting techniques. While classical methods (e.g. ARIMA) are long-standing approaches to forecasting, deep neural networks (e.g. NHITS) have recently shown state-of-the-art forecasting performance in benchmark datasets. We conducted extensive experiments that show NHITS generally performs best, but its superiority varies with forecasting conditions. For instance, concerning the forecasting horizon, NHITS only outperforms classical approaches for multi-step ahead forecasting. Another relevant insight is that, when dealing with anomalies, NHITS is outperformed by methods such as Theta. These findings highlight the importance of aspect-based model evaluation.

Related papers

• Loss Shaping Constraints for Long-Term Time Series Forecasting [79.3533114027664]
  We present a Constrained Learning approach for long-term time series forecasting that respects a user-defined upper bound on the loss at each time-step. We propose a practical Primal-Dual algorithm to tackle it, and aims to demonstrate that it exhibits competitive average performance in time series benchmarks, while shaping the errors across the predicted window.
  arXiv  Detail & Related papers  (2024-02-14T18:20:44Z)
• Predictive Churn with the Set of Good Models [64.05949860750235]
  We study the effect of conflicting predictions over the set of near-optimal machine learning models. We present theoretical results on the expected churn between models within the Rashomon set. We show how our approach can be used to better anticipate, reduce, and avoid churn in consumer-facing applications.
  arXiv  Detail & Related papers  (2024-02-12T16:15:25Z)
• ExtremeCast: Boosting Extreme Value Prediction for Global Weather Forecast [57.6987191099507]
  We introduce Exloss, a novel loss function that performs asymmetric optimization and highlights extreme values to obtain accurate extreme weather forecast. We also introduce ExBooster, which captures the uncertainty in prediction outcomes by employing multiple random samples. Our solution can achieve state-of-the-art performance in extreme weather prediction, while maintaining the overall forecast accuracy comparable to the top medium-range forecast models.
  arXiv  Detail & Related papers  (2024-02-02T10:34:13Z)
• Counterfactual Explanations for Time Series Forecasting [14.03870816983583]
  We formulate the novel problem of counterfactual generation for time series forecasting, and propose an algorithm, called ForecastCF. ForecastCF solves the problem by applying gradient-based perturbations to the original time series. Our results show that ForecastCF outperforms the baseline in terms of counterfactual validity and data manifold closeness.
  arXiv  Detail & Related papers  (2023-10-12T08:51:59Z)
• Forecasting inflation using disaggregates and machine learning [0.0]
  We consider different disaggregation levels for inflation and employ a range of traditional time series techniques as well as linear and nonlinear machine learning (ML) models to deal with a larger number of predictors. For many forecast horizons, the aggregation of disaggregated forecasts performs just as well survey-based expectations and models that generate forecasts using the aggregate directly. Our results reinforce the benefits of using models in a data-rich environment for inflation forecasting, including aggregating disaggregated forecasts from ML techniques.
  arXiv  Detail & Related papers  (2023-08-22T04:01:40Z)
• Forecast reconciliation for vaccine supply chain optimization [61.13962963550403]
  Vaccine supply chain optimization can benefit from hierarchical time series forecasting. Forecasts of different hierarchy levels become incoherent when higher levels do not match the sum of the lower levels forecasts. We tackle the vaccine sale forecasting problem by modeling sales data from GSK between 2010 and 2021 as a hierarchical time series.
  arXiv  Detail & Related papers  (2023-05-02T14:34:34Z)
• Prediction-Oriented Bayesian Active Learning [51.426960808684655]
  Expected predictive information gain (EPIG) is an acquisition function that measures information gain in the space of predictions rather than parameters. EPIG leads to stronger predictive performance compared with BALD across a range of datasets and models.
  arXiv  Detail & Related papers  (2023-04-17T10:59:57Z)
• Evaluation of Time-Series Forecasting Models for Chickenpox Cases Estimation in Hungary [0.0]
  We use time-series forecasting techniques to model and predict the future incidence of chickenpox. We implement and simulate multiple models and data preprocessing techniques on a Hungary-collected dataset.
  arXiv  Detail & Related papers  (2022-09-28T14:27:07Z)
• Long Range Probabilistic Forecasting in Time-Series using High Order Statistics [19.12411040726229]
  We present a novel probabilistic forecasting method that produces forecasts coherent in terms of base level and predicted aggregate statistics. We show that our method improves forecast performance across both base level and unseen aggregates post inference on real datasets ranging three diverse domains.
  arXiv  Detail & Related papers  (2021-11-05T11:10:35Z)
• Back2Future: Leveraging Backfill Dynamics for Improving Real-time Predictions in Future [73.03458424369657]
  In real-time forecasting in public health, data collection is a non-trivial and demanding task. 'Backfill' phenomenon and its effect on model performance has been barely studied in the prior literature. We formulate a novel problem and neural framework Back2Future that aims to refine a given model's predictions in real-time.
  arXiv  Detail & Related papers  (2021-06-08T14:48:20Z)
• Learning Prediction Intervals for Model Performance [1.433758865948252]
  We propose a method to compute prediction intervals for model performance. We evaluate our approach across a wide range of drift conditions and show substantial improvement over competitive baselines.
  arXiv  Detail & Related papers  (2020-12-15T21:32:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.