Global atmospheric data assimilation with multi-modal masked autoencoders
- URL: http://arxiv.org/abs/2407.11696v1
- Date: Tue, 16 Jul 2024 13:15:51 GMT
- Title: Global atmospheric data assimilation with multi-modal masked autoencoders
- Authors: Thomas J. Vandal, Kate Duffy, Daniel McDuff, Yoni Nachmany, Chris Hartshorn,
- Abstract summary: "EarthNet" is a multi-modal foundation model for data assimilation.
It learns to predict a global gap-filled atmospheric state solely from satellite observations.
It produces a global 0.16 degree reanalysis dataset of 3D atmospheric temperature and humidity.
- Score: 20.776143147372427
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Global data assimilation enables weather forecasting at all scales and provides valuable data for studying the Earth system. However, the computational demands of physics-based algorithms used in operational systems limits the volume and diversity of observations that are assimilated. Here, we present "EarthNet", a multi-modal foundation model for data assimilation that learns to predict a global gap-filled atmospheric state solely from satellite observations. EarthNet is trained as a masked autoencoder that ingests a 12 hour sequence of observations and learns to fill missing data from other sensors. We show that EarthNet performs a form of data assimilation producing a global 0.16 degree reanalysis dataset of 3D atmospheric temperature and humidity at a fraction of the time compared to operational systems. It is shown that the resulting reanalysis dataset reproduces climatology by evaluating a 1 hour forecast background state against observations. We also show that our 3D humidity predictions outperform MERRA-2 and ERA5 reanalyses by 10% to 60% between the middle troposphere and lower stratosphere (5 to 20 km altitude) and our 3D temperature and humidity are statistically equivalent to the Microwave integrated Retrieval System (MiRS) observations at nearly every level of the atmosphere. Our results indicate significant promise in using EarthNet for high-frequency data assimilation and global weather forecasting.
Related papers
- DeepExtremeCubes: Integrating Earth system spatio-temporal data for impact assessment of climate extremes [5.736700805381591]
Machine learning techniques show promise but require well-structured, high-quality, and curated analysis-ready datasets.
Here, we introduce the DeepExtremes database, tailored to map around heatwave and drought extreme impact.
It comprises over 40,000 spatially sampled small data cubes (i.e. minicubes) globally, with a spatial coverage of 2.5 by 2.5 km.
arXiv Detail & Related papers (2024-06-26T08:53:26Z) - Generative Data Assimilation of Sparse Weather Station Observations at Kilometer Scales [5.453657018459705]
We demonstrate the viability of score-based data assimilation in the context of realistically complex km-scale weather.
By incorporating observations from 40 weather stations, 10% lower RMSEs on left-out stations are attained.
It is a ripe time to explore extensions that combine increasingly ambitious regional state generators with an increasing set of in situ, ground-based, and satellite remote sensing data streams.
arXiv Detail & Related papers (2024-06-19T10:28:11Z) - Observation-Guided Meteorological Field Downscaling at Station Scale: A
Benchmark and a New Method [66.80344502790231]
We extend meteorological downscaling to arbitrary scattered station scales and establish a new benchmark and dataset.
Inspired by data assimilation techniques, we integrate observational data into the downscaling process, providing multi-scale observational priors.
Our proposed method outperforms other specially designed baseline models on multiple surface variables.
arXiv Detail & Related papers (2024-01-22T14:02:56Z) - DiffDA: a Diffusion Model for Weather-scale Data Assimilation [19.336483240566142]
We propose DiffDA as a denoising diffusion model capable of assimilating atmospheric variables using predicted states and sparse observations.
Acknowledging the similarity between a weather forecast model and a denoising diffusion model dedicated to weather applications, we adapt the pretrained GraphCast neural network as the backbone of the diffusion model.
arXiv Detail & Related papers (2024-01-11T14:11:12Z) - Towards an end-to-end artificial intelligence driven global weather forecasting system [57.5191940978886]
We present an AI-based data assimilation model, i.e., Adas, for global weather variables.
We demonstrate that Adas can assimilate global observations to produce high-quality analysis, enabling the system operate stably for long term.
We are the first to apply the methods to real-world scenarios, which is more challenging and has considerable practical application potential.
arXiv Detail & Related papers (2023-12-18T09:05:28Z) - Residual Diffusion Modeling for Km-scale Atmospheric Downscaling [51.061954281398116]
A cost-effective downscaling model is trained from a high-resolution 2-km weather model over Taiwan.
textitCorrDiff exhibits skillful RMSE and CRPS and faithfully recovers spectra and distributions even for extremes.
Downscaling global forecasts successfully retains many of these benefits, foreshadowing the potential of end-to-end, global-to-km-scales machine learning weather predictions.
arXiv Detail & Related papers (2023-09-24T19:57:22Z) - ClimaX: A foundation model for weather and climate [51.208269971019504]
ClimaX is a deep learning model for weather and climate science.
It can be pre-trained with a self-supervised learning objective on climate datasets.
It can be fine-tuned to address a breadth of climate and weather tasks.
arXiv Detail & Related papers (2023-01-24T23:19:01Z) - Pangu-Weather: A 3D High-Resolution Model for Fast and Accurate Global
Weather Forecast [91.9372563527801]
We present Pangu-Weather, a deep learning based system for fast and accurate global weather forecast.
For the first time, an AI-based method outperforms state-of-the-art numerical weather prediction (NWP) methods in terms of accuracy.
Pangu-Weather supports a wide range of downstream forecast scenarios, including extreme weather forecast and large-member ensemble forecast in real-time.
arXiv Detail & Related papers (2022-11-03T17:19:43Z) - Forecasting Global Weather with Graph Neural Networks [0.0]
We present a data-driven approach for forecasting global weather using graph neural networks.
The system learns to step forward the current 3D atmospheric state by six hours, and multiple steps are chained together to produce skillful forecasts going out several days into the future.
arXiv Detail & Related papers (2022-02-15T17:02:07Z) - Lidar Light Scattering Augmentation (LISA): Physics-based Simulation of
Adverse Weather Conditions for 3D Object Detection [60.89616629421904]
Lidar-based object detectors are critical parts of the 3D perception pipeline in autonomous navigation systems such as self-driving cars.
They are sensitive to adverse weather conditions such as rain, snow and fog due to reduced signal-to-noise ratio (SNR) and signal-to-background ratio (SBR)
arXiv Detail & Related papers (2021-07-14T21:10:47Z) - EarthNet2021: A novel large-scale dataset and challenge for forecasting
localized climate impacts [12.795776149170978]
Large Earth observation datasets now enable us to create machine learning models capable of translating coarse weather information into high-resolution Earth surface forecasts.
We define high-resolution Earth surface forecasting as video prediction of satellite imagery conditional on mesoscale weather forecasts.
We introduce EarthNet 2021, a new curated dataset containing target-temporal Sentinel 2 satellite imagery at 20 m resolution, matched with high-resolution topography and mesoscale (1.28 km) weather variables.
arXiv Detail & Related papers (2020-12-11T11:21:00Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.