TENT: Tensorized Encoder Transformer for Temperature Forecasting

• URL: http://arxiv.org/abs/2106.14742v1
• Date: Mon, 28 Jun 2021 14:17:22 GMT
• Title: TENT: Tensorized Encoder Transformer for Temperature Forecasting
• Authors: Onur Bilgin, Pawe{\l} M\k{a}ka, Thomas Vergutz and Siamak Mehrkanoon
• Abstract summary: We introduce a new model based on the Transformer architecture for weather forecasting. We show that compared to the original transformer and 3D convolutional neural networks, the proposed TENT model can better model the underlying complex pattern of weather data. Experiments on two real-life weather datasets are performed.
• Score: 3.498371632913735
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Reliable weather forecasting is of great importance in science, business and society. The best performing data-driven models for weather prediction tasks rely on recurrent or convolutional neural networks, where some of which incorporate attention mechanisms. In this work, we introduce a new model based on the Transformer architecture for weather forecasting. The proposed Tensorial Encoder Transformer (TENT) model is equipped with tensorial attention and thus it exploits the spatiotemporal structure of weather data by processing it in multidimensional tensorial format. We show that compared to the encoder part of the original transformer and 3D convolutional neural networks, the proposed TENT model can better model the underlying complex pattern of weather data for the studied temperature prediction task. Experiments on two real-life weather datasets are performed. The datasets consist of historical measurements from USA, Canada and European cities. The first dataset contains hourly measurements of weather attributes for 30 cities in USA and Canada from October 2012 to November 2017. The second dataset contains daily measurements of weather attributes of 18 cities across Europe from May 2005 to April 2020. We use attention scores calculated from our attention mechanism to shed light on the decision-making process of our model and have insight knowledge on the most important cities for the task.

Related papers

• FATE: Focal-modulated Attention Encoder for Temperature Prediction [0.0]
  Climate change is one of the major challenges of the twenty-first century. Traditional data-driven models often use recurrent neural networks (RNNs) but face limitations in parallelization. We introduce a novel approach based on the FocalNet Transformer architecture.
  arXiv  Detail & Related papers  (2024-08-21T04:40:18Z)
• WeatherFormer: A Pretrained Encoder Model for Learning Robust Weather Representations from Small Datasets [0.5735035463793009]
  WeatherFormer is a transformer encoder-based model designed to learn robust weather features from minimal observations. WeatherFormer was pretrained on a large pretraining dataset comprised of 39 years of satellite measurements across the Americas.
  arXiv  Detail & Related papers  (2024-05-22T17:43:46Z)
• Generalizing Weather Forecast to Fine-grained Temporal Scales via Physics-AI Hybrid Modeling [55.13352174687475]
  This paper proposes a physics-AI hybrid model (i.e., WeatherGFT) which Generalizes weather forecasts to Finer-grained Temporal scales. Specifically, we employ a carefully designed PDE kernel to simulate physical evolution on a small time scale. We introduce a lead time-aware training framework to promote the generalization of the model at different lead times.
  arXiv  Detail & Related papers  (2024-05-22T16:21:02Z)
• FengWu-4DVar: Coupling the Data-driven Weather Forecasting Model with 4D Variational Assimilation [67.20588721130623]
  We develop an AI-based cyclic weather forecasting system, FengWu-4DVar. FengWu-4DVar can incorporate observational data into the data-driven weather forecasting model. Experiments on the simulated observational dataset demonstrate that FengWu-4DVar is capable of generating reasonable analysis fields.
  arXiv  Detail & Related papers  (2023-12-16T02:07:56Z)
• Deep Learning for Day Forecasts from Sparse Observations [60.041805328514876]
  Deep neural networks offer an alternative paradigm for modeling weather conditions. MetNet-3 learns from both dense and sparse data sensors and makes predictions up to 24 hours ahead for precipitation, wind, temperature and dew point. MetNet-3 has a high temporal and spatial resolution, respectively, up to 2 minutes and 1 km as well as a low operational latency.
  arXiv  Detail & Related papers  (2023-06-06T07:07:54Z)
• GraphCast: Learning skillful medium-range global weather forecasting [107.40054095223779]
  We introduce a machine learning-based method called "GraphCast", which can be trained directly from reanalysis data. It predicts hundreds of weather variables, over 10 days at 0.25 degree resolution globally, in under one minute. We show that GraphCast significantly outperforms the most accurate operational deterministic systems on 90% of 1380 verification targets.
  arXiv  Detail & Related papers  (2022-12-24T18:15:39Z)
• Pangu-Weather: A 3D High-Resolution Model for Fast and Accurate Global Weather Forecast [91.9372563527801]
  We present Pangu-Weather, a deep learning based system for fast and accurate global weather forecast. For the first time, an AI-based method outperforms state-of-the-art numerical weather prediction (NWP) methods in terms of accuracy. Pangu-Weather supports a wide range of downstream forecast scenarios, including extreme weather forecast and large-member ensemble forecast in real-time.
  arXiv  Detail & Related papers  (2022-11-03T17:19:43Z)
• A Novel Transformer Network with Shifted Window Cross-Attention for Spatiotemporal Weather Forecasting [5.414308305392762]
  We tackle the challenge of weather forecasting using a video transformer network. Vision transformer architectures have been explored in various applications. We propose the use of Video Swin-Transformer, coupled with a dedicated augmentation scheme.
  arXiv  Detail & Related papers  (2022-08-02T05:04:53Z)
• Multistream Graph Attention Networks for Wind Speed Forecasting [4.644923443649426]
  This paper presents a new model for wind speed prediction based on Graph Attention Networks (GAT) In particular, the proposed model extends GAT architecture by equipping it with a learnable adjacency matrix. We show that in comparison to previous architectures used for wind speed prediction, the proposed model is able to better learn the complex input-output relationships of the weather data.
  arXiv  Detail & Related papers  (2021-08-16T12:58:26Z)
• Wind speed prediction using multidimensional convolutional neural networks [5.228711636020665]
  This paper introduces a model based on convolutional neural networks (CNNs) for wind speed prediction tasks. We show that compared to classical CNN-based models, the proposed model is able to better characterise the wind data.
  arXiv  Detail & Related papers  (2020-07-04T20:48:41Z)
• A generative adversarial network approach to (ensemble) weather prediction [91.3755431537592]
  We use a conditional deep convolutional generative adversarial network to predict the geopotential height of the 500 hPa pressure level, the two-meter temperature and the total precipitation for the next 24 hours over Europe. The proposed models are trained on 4 years of ERA5 reanalysis data from 2015-2018 with the goal to predict the associated meteorological fields in 2019.
  arXiv  Detail & Related papers  (2020-06-13T20:53:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.