WeatherFormer: A Pretrained Encoder Model for Learning Robust Weather Representations from Small Datasets

• URL: http://arxiv.org/abs/2405.17455v1
• Date: Wed, 22 May 2024 17:43:46 GMT
• Title: WeatherFormer: A Pretrained Encoder Model for Learning Robust Weather Representations from Small Datasets
• Authors: Adib Hasan, Mardavij Roozbehani, Munther Dahleh,
• Abstract summary: WeatherFormer is a transformer encoder-based model designed to learn robust weather features from minimal observations. WeatherFormer was pretrained on a large pretraining dataset comprised of 39 years of satellite measurements across the Americas.
• Score: 0.5735035463793009
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This paper introduces WeatherFormer, a transformer encoder-based model designed to learn robust weather features from minimal observations. It addresses the challenge of modeling complex weather dynamics from small datasets, a bottleneck for many prediction tasks in agriculture, epidemiology, and climate science. WeatherFormer was pretrained on a large pretraining dataset comprised of 39 years of satellite measurements across the Americas. With a novel pretraining task and fine-tuning, WeatherFormer achieves state-of-the-art performance in county-level soybean yield prediction and influenza forecasting. Technical innovations include a unique spatiotemporal encoding that captures geographical, annual, and seasonal variations, adapting the transformer architecture to continuous weather data, and a pretraining strategy to learn representations that are robust to missing weather features. This paper for the first time demonstrates the effectiveness of pretraining large transformer encoder models for weather-dependent applications across multiple domains.

Related papers

• LightWeather: Harnessing Absolute Positional Encoding to Efficient and Scalable Global Weather Forecasting [21.048535830456363]
  We show that absolute positional encoding is what really works in Transformer-based weather forecasting models. We propose LightWeather, a lightweight and effective model for station-of-based global weather forecasting.
  arXiv  Detail & Related papers  (2024-08-19T04:23:40Z)
• Generalizing Weather Forecast to Fine-grained Temporal Scales via Physics-AI Hybrid Modeling [55.13352174687475]
  This paper proposes a physics-AI hybrid model (i.e., WeatherGFT) which Generalizes weather forecasts to Finer-grained Temporal scales. Specifically, we employ a carefully designed PDE kernel to simulate physical evolution on a small time scale. We introduce a lead time-aware training framework to promote the generalization of the model at different lead times.
  arXiv  Detail & Related papers  (2024-05-22T16:21:02Z)
• ExtremeCast: Boosting Extreme Value Prediction for Global Weather Forecast [57.6987191099507]
  We introduce Exloss, a novel loss function that performs asymmetric optimization and highlights extreme values to obtain accurate extreme weather forecast. We also introduce ExBooster, which captures the uncertainty in prediction outcomes by employing multiple random samples. Our solution can achieve state-of-the-art performance in extreme weather prediction, while maintaining the overall forecast accuracy comparable to the top medium-range forecast models.
  arXiv  Detail & Related papers  (2024-02-02T10:34:13Z)
• Scaling transformer neural networks for skillful and reliable medium-range weather forecasting [23.249955524044392]
  We introduce Stormer, a state-of-the-art performance on weather forecasting with minimal changes to the standard transformer backbone. At the core of Stormer is a randomized forecasting objective that trains the model to forecast the weather dynamics over varying time intervals. On WeatherBench 2, Stormer performs competitively at short to medium-range forecasts and outperforms current methods beyond 7 days.
  arXiv  Detail & Related papers  (2023-12-06T19:46:06Z)
• Learning Robust Precipitation Forecaster by Temporal Frame Interpolation [65.5045412005064]
  We develop a robust precipitation forecasting model that demonstrates resilience against spatial-temporal discrepancies. Our approach has led to significant improvements in forecasting precision, culminating in our model securing textit1st place in the transfer learning leaderboard of the textitWeather4cast'23 competition.
  arXiv  Detail & Related papers  (2023-11-30T08:22:08Z)
• Exploring the Application of Large-scale Pre-trained Models on Adverse Weather Removal [97.53040662243768]
  We propose a CLIP embedding module to make the network handle different weather conditions adaptively. This module integrates the sample specific weather prior extracted by CLIP image encoder together with the distribution specific information learned by a set of parameters.
  arXiv  Detail & Related papers  (2023-06-15T10:06:13Z)
• ClimaX: A foundation model for weather and climate [51.208269971019504]
  ClimaX is a deep learning model for weather and climate science. It can be pre-trained with a self-supervised learning objective on climate datasets. It can be fine-tuned to address a breadth of climate and weather tasks.
  arXiv  Detail & Related papers  (2023-01-24T23:19:01Z)
• GraphCast: Learning skillful medium-range global weather forecasting [107.40054095223779]
  We introduce a machine learning-based method called "GraphCast", which can be trained directly from reanalysis data. It predicts hundreds of weather variables, over 10 days at 0.25 degree resolution globally, in under one minute. We show that GraphCast significantly outperforms the most accurate operational deterministic systems on 90% of 1380 verification targets.
  arXiv  Detail & Related papers  (2022-12-24T18:15:39Z)
• TENT: Tensorized Encoder Transformer for Temperature Forecasting [3.498371632913735]
  We introduce a new model based on the Transformer architecture for weather forecasting. We show that compared to the original transformer and 3D convolutional neural networks, the proposed TENT model can better model the underlying complex pattern of weather data. Experiments on two real-life weather datasets are performed.
  arXiv  Detail & Related papers  (2021-06-28T14:17:22Z)
• Smart Weather Forecasting Using Machine Learning:A Case Study in Tennessee [2.9477900773805032]
  We present a weather prediction technique that utilizes historical data from multiple weather stations to train simple machine learning models. The accuracy of the models is good enough to be used alongside the current state-of-the-art techniques.
  arXiv  Detail & Related papers  (2020-08-25T02:41:32Z)
• SmaAt-UNet: Precipitation Nowcasting using a Small Attention-UNet Architecture [5.28539620288341]
  We show that it is possible to produce an accurate precipitation nowcast using a data-driven neural network approach. We evaluate our approaches on a real-life datasets using precipitation maps from the region of the Netherlands and binary images of cloud coverage of France.
  arXiv  Detail & Related papers  (2020-07-08T20:33:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.