STC-ViT: Spatio Temporal Continuous Vision Transformer for Weather Forecasting
- URL: http://arxiv.org/abs/2402.17966v2
- Date: Fri, 24 May 2024 00:19:33 GMT
- Title: STC-ViT: Spatio Temporal Continuous Vision Transformer for Weather Forecasting
- Authors: Hira Saleem, Flora Salim, Cormac Purcell,
- Abstract summary: Transformer models have shown remarkable potential in weather forecasting achieving state-of-the-art results.
STC-ViT incorporates continuous Neural ODE layers with multi-head attention mechanism to learn the continuous weather evolution over time.
We evaluate STC-ViT against a operational Numerical Weather Prediction (NWP) model and several deep learning based weather forecasting models.
- Score: 0.0
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Operational weather forecasting system relies on computationally expensive physics-based models. Recently, transformer based models have shown remarkable potential in weather forecasting achieving state-of-the-art results. However, transformers are discrete models which limit their ability to learn the continuous spatio-temporal features of the dynamical weather system. We address this issue with STC-ViT, a Spatio-Temporal Continuous Vision Transformer for weather forecasting. STC-ViT incorporates the continuous time Neural ODE layers with multi-head attention mechanism to learn the continuous weather evolution over time. The attention mechanism is encoded as a differentiable function in the transformer architecture to model the complex weather dynamics. We evaluate STC-ViT against a operational Numerical Weather Prediction (NWP) model and several deep learning based weather forecasting models. STC-ViT performs competitively with current data-driven methods in global forecasting while only being trained at lower resolution data and with less compute power.
Related papers
- WeatherFormer: A Pretrained Encoder Model for Learning Robust Weather Representations from Small Datasets [0.5735035463793009]
WeatherFormer is a transformer encoder-based model designed to learn robust weather features from minimal observations.
WeatherFormer was pretrained on a large pretraining dataset comprised of 39 years of satellite measurements across the Americas.
arXiv Detail & Related papers (2024-05-22T17:43:46Z) - Generalizing Weather Forecast to Fine-grained Temporal Scales via Physics-AI Hybrid Modeling [55.13352174687475]
This paper proposes a physics-AI hybrid model (i.e., WeatherGFT) which Generalizes weather forecasts to Finer-grained Temporal scales.
Specifically, we employ a carefully designed PDE kernel to simulate physical evolution on a small time scale.
We introduce a lead time-aware training framework to promote the generalization of the model at different lead times.
arXiv Detail & Related papers (2024-05-22T16:21:02Z) - ClimODE: Climate and Weather Forecasting with Physics-informed Neural ODEs [14.095897879222676]
We present ClimODE, a continuous-time process that implements key principle of statistical mechanics.
ClimODE models precise weather evolution with value-conserving dynamics, learning global weather transport as a neural flow.
Our approach outperforms existing data-driven methods in global, regional forecasting with an order of magnitude smaller parameterization.
arXiv Detail & Related papers (2024-04-15T06:38:21Z) - Rethinking Urban Mobility Prediction: A Super-Multivariate Time Series
Forecasting Approach [71.67506068703314]
Long-term urban mobility predictions play a crucial role in the effective management of urban facilities and services.
Traditionally, urban mobility data has been structured as videos, treating longitude and latitude as fundamental pixels.
In our research, we introduce a fresh perspective on urban mobility prediction.
Instead of oversimplifying urban mobility data as traditional video data, we regard it as a complex time series.
arXiv Detail & Related papers (2023-12-04T07:39:05Z) - CARD: Channel Aligned Robust Blend Transformer for Time Series
Forecasting [50.23240107430597]
We design a special Transformer, i.e., Channel Aligned Robust Blend Transformer (CARD for short), that addresses key shortcomings of CI type Transformer in time series forecasting.
First, CARD introduces a channel-aligned attention structure that allows it to capture both temporal correlations among signals.
Second, in order to efficiently utilize the multi-scale knowledge, we design a token blend module to generate tokens with different resolutions.
Third, we introduce a robust loss function for time series forecasting to alleviate the potential overfitting issue.
arXiv Detail & Related papers (2023-05-20T05:16:31Z) - ClimaX: A foundation model for weather and climate [51.208269971019504]
ClimaX is a deep learning model for weather and climate science.
It can be pre-trained with a self-supervised learning objective on climate datasets.
It can be fine-tuned to address a breadth of climate and weather tasks.
arXiv Detail & Related papers (2023-01-24T23:19:01Z) - Towards Long-Term Time-Series Forecasting: Feature, Pattern, and
Distribution [57.71199089609161]
Long-term time-series forecasting (LTTF) has become a pressing demand in many applications, such as wind power supply planning.
Transformer models have been adopted to deliver high prediction capacity because of the high computational self-attention mechanism.
We propose an efficient Transformerbased model, named Conformer, which differentiates itself from existing methods for LTTF in three aspects.
arXiv Detail & Related papers (2023-01-05T13:59:29Z) - Physics-Inspired Temporal Learning of Quadrotor Dynamics for Accurate
Model Predictive Trajectory Tracking [76.27433308688592]
Accurately modeling quadrotor's system dynamics is critical for guaranteeing agile, safe, and stable navigation.
We present a novel Physics-Inspired Temporal Convolutional Network (PI-TCN) approach to learning quadrotor's system dynamics purely from robot experience.
Our approach combines the expressive power of sparse temporal convolutions and dense feed-forward connections to make accurate system predictions.
arXiv Detail & Related papers (2022-06-07T13:51:35Z) - Numerical Weather Forecasting using Convolutional-LSTM with Attention
and Context Matcher Mechanisms [10.759556555869798]
We introduce a novel deep learning architecture for forecasting high-resolution weather data.
Our Weather Model achieves significant performance improvements compared to baseline deep learning models.
arXiv Detail & Related papers (2021-02-01T08:30:42Z) - Improving data-driven global weather prediction using deep convolutional
neural networks on a cubed sphere [7.918783985810551]
We present a significantly-improved data-driven global weather forecasting framework using a deep convolutional neural network (CNN)
New developments in this framework include an offline volume-conservative mapping to a cubed-sphere grid.
Our model is able to learn to forecast complex surface temperature patterns from few input atmospheric state variables.
arXiv Detail & Related papers (2020-03-15T19:57:34Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.