Balancing Computational Efficiency and Forecast Error in Machine Learning-based Time-Series Forecasting: Insights from Live Experiments on Meteorological Nowcasting

• URL: http://arxiv.org/abs/2309.15207v1
• Date: Tue, 26 Sep 2023 19:10:00 GMT
• Title: Balancing Computational Efficiency and Forecast Error in Machine Learning-based Time-Series Forecasting: Insights from Live Experiments on Meteorological Nowcasting
• Authors: Elin T\"ornquist, Wagner Costa Santos, Timothy Pogue, Nicholas Wingle, Robert A. Caulk
• Abstract summary: This paper addresses the relationship between computational cost and forecast error using meteorological nowcasting. During a 5-day live experiment, 4000 data sources were streamed for training and inferencing 144 models per hour. Results indicate that using the Variance Horizon reduced computational usage by more than 50%, while increasing between 0-15% in error.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Machine learning for time-series forecasting remains a key area of research. Despite successful application of many machine learning techniques, relating computational efficiency to forecast error remains an under-explored domain. This paper addresses this topic through a series of real-time experiments to quantify the relationship between computational cost and forecast error using meteorological nowcasting as an example use-case. We employ a variety of popular regression techniques (XGBoost, FC-MLP, Transformer, and LSTM) for multi-horizon, short-term forecasting of three variables (temperature, wind speed, and cloud cover) for multiple locations. During a 5-day live experiment, 4000 data sources were streamed for training and inferencing 144 models per hour. These models were parameterized to explore forecast error for two computational cost minimization methods: a novel auto-adaptive data reduction technique (Variance Horizon) and a performance-based concept drift-detection mechanism. Forecast error of all model variations were benchmarked in real-time against a state-of-the-art numerical weather prediction model. Performance was assessed using classical and novel evaluation metrics. Results indicate that using the Variance Horizon reduced computational usage by more than 50\%, while increasing between 0-15\% in error. Meanwhile, performance-based retraining reduced computational usage by up to 90\% while \emph{also} improving forecast error by up to 10\%. Finally, the combination of both the Variance Horizon and performance-based retraining outperformed other model configurations by up to 99.7\% when considering error normalized to computational usage.

Related papers

• UNet with Axial Transformer : A Neural Weather Model for Precipitation Nowcasting [0.06906005491572399]
  We develop a novel method that employs Transformer-based machine learning models to forecast precipitation. This paper represents an initial research on the dataset used in the domain of next frame prediciton.
  arXiv  Detail & Related papers  (2025-04-28T01:20:30Z)
• Generalizing Weather Forecast to Fine-grained Temporal Scales via Physics-AI Hybrid Modeling [55.13352174687475]
  This paper proposes a physics-AI hybrid model (i.e., WeatherGFT) which Generalizes weather forecasts to Finer-grained Temporal scales. Specifically, we employ a carefully designed PDE kernel to simulate physical evolution on a small time scale. We introduce a lead time-aware training framework to promote the generalization of the model at different lead times.
  arXiv  Detail & Related papers  (2024-05-22T16:21:02Z)
• Data Assimilation with Machine Learning Surrogate Models: A Case Study with FourCastNet [10.773673764125439]
  This paper investigates online weather prediction using machine learning surrogates supplemented with partial and noisy observations. We empirically demonstrate and theoretically justify that, despite the long-time instability of the surrogates, filtering estimates can remain accurate in the long-time horizon.
  arXiv  Detail & Related papers  (2024-05-21T20:06:12Z)
• CaFA: Global Weather Forecasting with Factorized Attention on Sphere [7.687215328455751]
  We propose a factorized-attention-based model tailored for spherical geometries to mitigate this issue. The deterministic forecasting accuracy of the proposed model on $1.5circ$ and 0-7 days' lead time is on par with state-of-the-art purely data-driven machine learning weather prediction models.
  arXiv  Detail & Related papers  (2024-05-12T23:18:14Z)
• ExtremeCast: Boosting Extreme Value Prediction for Global Weather Forecast [57.6987191099507]
  We introduce Exloss, a novel loss function that performs asymmetric optimization and highlights extreme values to obtain accurate extreme weather forecast. We also introduce ExBooster, which captures the uncertainty in prediction outcomes by employing multiple random samples. Our solution can achieve state-of-the-art performance in extreme weather prediction, while maintaining the overall forecast accuracy comparable to the top medium-range forecast models.
  arXiv  Detail & Related papers  (2024-02-02T10:34:13Z)
• Learning Robust Precipitation Forecaster by Temporal Frame Interpolation [65.5045412005064]
  We develop a robust precipitation forecasting model that demonstrates resilience against spatial-temporal discrepancies. Our approach has led to significant improvements in forecasting precision, culminating in our model securing textit1st place in the transfer learning leaderboard of the textitWeather4cast'23 competition.
  arXiv  Detail & Related papers  (2023-11-30T08:22:08Z)
• ClimaX: A foundation model for weather and climate [51.208269971019504]
  ClimaX is a deep learning model for weather and climate science. It can be pre-trained with a self-supervised learning objective on climate datasets. It can be fine-tuned to address a breadth of climate and weather tasks.
  arXiv  Detail & Related papers  (2023-01-24T23:19:01Z)
• SpectraNet: Multivariate Forecasting and Imputation under Distribution Shifts and Missing Data [40.21502451136054]
  SpectraNet is a novel time-series forecasting model that dynamically infers temporal dynamics and correlations on the recent observed history. A Convolution Neural Network maps the learned representation by sequentially mixing its components and refining the output. Our proposed approach can simultaneously produce forecasts and interpolate past observations and can, therefore, greatly simplify production systems.
  arXiv  Detail & Related papers  (2022-10-22T18:07:31Z)
• SwinVRNN: A Data-Driven Ensemble Forecasting Model via Learned Distribution Perturbation [16.540748935603723]
  We propose a Swin Transformer-based Variational Recurrent Neural Network (SwinVRNN), which is a weather forecasting model combining a SwinRNN predictor with a perturbation module. SwinVRNN surpasses operational ECMWF Integrated Forecasting System (IFS) on surface variables of 2-m temperature and 6-hourly total precipitation at all lead times up to five days.
  arXiv  Detail & Related papers  (2022-05-26T05:11:58Z)
• Physics Informed Shallow Machine Learning for Wind Speed Prediction [66.05661813632568]
  We analyze a massive dataset of wind measured from anemometers located at 10 m height in 32 locations in Italy. We train supervised learning algorithms using the past history of wind to predict its value at a future time. We find that the optimal design as well as its performance vary with the location.
  arXiv  Detail & Related papers  (2022-04-01T14:55:10Z)
• Adjusting for Autocorrelated Errors in Neural Networks for Time Series Regression and Forecasting [10.659189276058948]
  We learn the autocorrelation coefficient jointly with the model parameters in order to adjust for autocorrelated errors. For time series regression, large-scale experiments indicate that our method outperforms the Prais-Winsten method. Results across a wide range of real-world datasets show that our method enhances performance in almost all cases.
  arXiv  Detail & Related papers  (2021-01-28T04:25:51Z)
• Predicting Training Time Without Training [120.92623395389255]
  We tackle the problem of predicting the number of optimization steps that a pre-trained deep network needs to converge to a given value of the loss function. We leverage the fact that the training dynamics of a deep network during fine-tuning are well approximated by those of a linearized model. We are able to predict the time it takes to fine-tune a model to a given loss without having to perform any training.
  arXiv  Detail & Related papers  (2020-08-28T04:29:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.