A Practical Probabilistic Benchmark for AI Weather Models
- URL: http://arxiv.org/abs/2401.15305v1
- Date: Sat, 27 Jan 2024 05:53:16 GMT
- Title: A Practical Probabilistic Benchmark for AI Weather Models
- Authors: Noah D. Brenowitz and Yair Cohen and Jaideep Pathak and Ankur Mahesh
and Boris Bonev and Thorsten Kurth and Dale R. Durran and Peter Harrington
and Michael S. Pritchard
- Abstract summary: We show that two leading AI weather models, i.e. GraphCast and Pangu, are tied on the probabilistic CRPS metric.
We also reveal how multiple time-step loss functions, which many data-driven weather models have employed, are counter-productive.
- Score: 0.8278356279004184
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Since the weather is chaotic, forecasts aim to predict the distribution of
future states rather than make a single prediction. Recently, multiple data
driven weather models have emerged claiming breakthroughs in skill. However,
these have mostly been benchmarked using deterministic skill scores, and little
is known about their probabilistic skill. Unfortunately, it is hard to fairly
compare AI weather models in a probabilistic sense, since variations in choice
of ensemble initialization, definition of state, and noise injection
methodology become confounding. Moreover, even obtaining ensemble forecast
baselines is a substantial engineering challenge given the data volumes
involved. We sidestep both problems by applying a decades-old idea -- lagged
ensembles -- whereby an ensemble can be constructed from a moderately-sized
library of deterministic forecasts. This allows the first parameter-free
intercomparison of leading AI weather models' probabilistic skill against an
operational baseline. The results reveal that two leading AI weather models,
i.e. GraphCast and Pangu, are tied on the probabilistic CRPS metric even though
the former outperforms the latter in deterministic scoring. We also reveal how
multiple time-step loss functions, which many data-driven weather models have
employed, are counter-productive: they improve deterministic metrics at the
cost of increased dissipation, deteriorating probabilistic skill. This is
confirmed through ablations applied to a spherical Fourier Neural Operator
(SFNO) approach to AI weather forecasting. Separate SFNO ablations modulating
effective resolution reveal it has a useful effect on ensemble dispersion
relevant to achieving good ensemble calibration. We hope these and forthcoming
insights from lagged ensembles can help guide the development of AI weather
forecasts and have thus shared the diagnostic code.
Related papers
- Lightning-Fast Thunderstorm Warnings: Predicting Severe Convective Environments with Global Neural Weather Models [0.08271752505511926]
Recently released suite of AI weather models can produce multi-day, medium-range forecasts within seconds.
Traditional AI model evaluation predominantly targets global scores on single levels.
With a focus on the convective season of global hotspots in 2020, we assess the skill of three top-performing AI models.
arXiv Detail & Related papers (2024-06-13T07:46:03Z) - Probabilistic Weather Forecasting with Hierarchical Graph Neural Networks [17.64833210797824]
We propose a probabilistic weather forecasting model called Graph-EFM.
The model combines a flexible latent-variable formulation with the successful graph-based forecasting framework.
Ensemble forecasts from Graph-EFM achieve equivalent or lower errors than comparable deterministic models.
arXiv Detail & Related papers (2024-06-07T09:01:25Z) - Uncertainty quantification for data-driven weather models [0.0]
We study and compare uncertainty quantification methods to generate probabilistic weather forecasts from a state-of-the-art deterministic data-driven weather model, Pangu-Weather.
Specifically, we compare approaches for quantifying forecast uncertainty based on generating ensemble forecasts via perturbations to the initial conditions.
In a case study on medium-range forecasts of selected weather variables over Europe, the probabilistic forecasts obtained by using the Pangu-Weather model in concert with uncertainty quantification methods show promising results.
arXiv Detail & Related papers (2024-03-20T10:07:51Z) - ExtremeCast: Boosting Extreme Value Prediction for Global Weather Forecast [57.6987191099507]
We introduce Exloss, a novel loss function that performs asymmetric optimization and highlights extreme values to obtain accurate extreme weather forecast.
We also introduce a training-free extreme value enhancement strategy named ExEnsemble, which increases the variance of pixel values and improves the forecast robustness.
Our solution can achieve state-of-the-art performance in extreme weather prediction, while maintaining the overall forecast accuracy comparable to the top medium-range forecast models.
arXiv Detail & Related papers (2024-02-02T10:34:13Z) - Towards an end-to-end artificial intelligence driven global weather forecasting system [57.5191940978886]
We present an AI-based data assimilation model, i.e., Adas, for global weather variables.
We demonstrate that Adas can assimilate global observations to produce high-quality analysis, enabling the system operate stably for long term.
We are the first to apply the methods to real-world scenarios, which is more challenging and has considerable practical application potential.
arXiv Detail & Related papers (2023-12-18T09:05:28Z) - Human Trajectory Forecasting with Explainable Behavioral Uncertainty [63.62824628085961]
Human trajectory forecasting helps to understand and predict human behaviors, enabling applications from social robots to self-driving cars.
Model-free methods offer superior prediction accuracy but lack explainability, while model-based methods provide explainability but cannot predict well.
We show that BNSP-SFM achieves up to a 50% improvement in prediction accuracy, compared with 11 state-of-the-art methods.
arXiv Detail & Related papers (2023-07-04T16:45:21Z) - SEEDS: Emulation of Weather Forecast Ensembles with Diffusion Models [13.331224394143117]
Uncertainty quantification is crucial to decision-making.
dominant approach to representing uncertainty in weather forecasting is to generate an ensemble of forecasts.
We propose to amortize the computational cost by emulating these forecasts with deep generative diffusion models learned from historical data.
arXiv Detail & Related papers (2023-06-24T22:00:06Z) - Uncertainty estimation of pedestrian future trajectory using Bayesian
approximation [137.00426219455116]
Under dynamic traffic scenarios, planning based on deterministic predictions is not trustworthy.
The authors propose to quantify uncertainty during forecasting using approximation which deterministic approaches fail to capture.
The effect of dropout weights and long-term prediction on future state uncertainty has been studied.
arXiv Detail & Related papers (2022-05-04T04:23:38Z) - Complex Event Forecasting with Prediction Suffix Trees: Extended
Technical Report [70.7321040534471]
Complex Event Recognition (CER) systems have become popular in the past two decades due to their ability to "instantly" detect patterns on real-time streams of events.
There is a lack of methods for forecasting when a pattern might occur before such an occurrence is actually detected by a CER engine.
We present a formal framework that attempts to address the issue of Complex Event Forecasting.
arXiv Detail & Related papers (2021-09-01T09:52:31Z) - A framework for probabilistic weather forecast post-processing across
models and lead times using machine learning [3.1542695050861544]
We show how to bridge the gap between sets of separate forecasts from NWP models and the 'ideal' forecast for decision support.
We use Quantile Regression Forests to learn the error profile of each numerical model, and use these to apply empirically-derived probability distributions to forecasts.
Second, we combine these probabilistic forecasts using quantile averaging. Third, we interpolate between the aggregate quantiles in order to generate a full predictive distribution.
arXiv Detail & Related papers (2020-05-06T16:46:02Z) - Ambiguity in Sequential Data: Predicting Uncertain Futures with
Recurrent Models [110.82452096672182]
We propose an extension of the Multiple Hypothesis Prediction (MHP) model to handle ambiguous predictions with sequential data.
We also introduce a novel metric for ambiguous problems, which is better suited to account for uncertainties.
arXiv Detail & Related papers (2020-03-10T09:15:42Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.