SAFE: A Novel Approach to AI Weather Evaluation through Stratified Assessments of Forecasts over Earth

• URL: http://arxiv.org/abs/2510.26099v1
• Date: Thu, 30 Oct 2025 03:22:55 GMT
• Title: SAFE: A Novel Approach to AI Weather Evaluation through Stratified Assessments of Forecasts over Earth
• Abstract summary: Stratified Assessments of Forecasts over Earth (SAFE) is a package for elucidating the stratified performance of a set of predictions made over Earth.<n>SAFE integrates various data domains to stratify by different attributes associated with geospatial gridpoints.<n>To demonstrate its importance, we utilize SAFE to benchmark a zoo of state-of-the-art AI-based weather prediction models.
• Score: 18.705966042259405
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The dominant paradigm in machine learning is to assess model performance based on average loss across all samples in some test set. This amounts to averaging performance geospatially across the Earth in weather and climate settings, failing to account for the non-uniform distribution of human development and geography. We introduce Stratified Assessments of Forecasts over Earth (SAFE), a package for elucidating the stratified performance of a set of predictions made over Earth. SAFE integrates various data domains to stratify by different attributes associated with geospatial gridpoints: territory (usually country), global subregion, income, and landcover (land or water). This allows us to examine the performance of models for each individual stratum of the different attributes (e.g., the accuracy in every individual country). To demonstrate its importance, we utilize SAFE to benchmark a zoo of state-of-the-art AI-based weather prediction models, finding that they all exhibit disparities in forecasting skill across every attribute. We use this to seed a benchmark of model forecast fairness through stratification at different lead times for various climatic variables. By moving beyond globally-averaged metrics, we for the first time ask: where do models perform best or worst, and which models are most fair? To support further work in this direction, the SAFE package is open source and available at https://github.com/N-Masi/safe

Related papers

• EO-WM: A Physically Informed World Model for Probabilistic Earth Observation Forecasting [50.31500063307306]
  We introduce EO-WM, a video diffusion transformer for multispectral Earth Observation forecasting.<n> EO-WM incorporates a physically informed conditioning framework that represents meteorological forcing through a climatological baseline, weather anomalies, and cumulative physical stress signals.<n>Experiments show that EO-WM reduces the error in predicted Normalized Difference Vegetation Index (NDVI) decline amplitude by a relative 5.63% and improves directional hit rate by a relative 7.80%.
  arXiv  Detail & Related papers  (2026-06-25T16:53:30Z)
• Earth System Foundation Model (ESFM): A unified framework for heterogeneous data integration and forecasting [52.53742727060265]
  Foundation models (FMs) for the Earth system learn statistical relationships between physical variables across massive datasets.<n>Here, we introduce Earth System Foundation Model (ESFM), a fully open model building on the 3Dwin UNet backbone of the pioneering model.<n>We present findings using dense gridded data (ERA5, CMIP6), regionally masked dense data, sparse gridded MODIS satellite data, and station data.<n>Results demonstrate competitive or superior performance relative to state-of-the-art benchmarks.
  arXiv  Detail & Related papers  (2026-04-20T11:40:39Z)
• On Global Applicability and Location Transferability of Generative Deep Learning Models for Precipitation Downscaling [14.69597733830489]
  We evaluate the generalization performance of generative downscaling models across diverse regions.<n>A hierarchical location-based data split enables a systematic assessment of model performance across 15 regions around the world.
  arXiv  Detail & Related papers  (2025-12-01T08:24:40Z)
• GEO-Bench-2: From Performance to Capability, Rethinking Evaluation in Geospatial AI [52.13138825802668]
  GeoFMs are transforming Earth Observation, but evaluation lacks standardized protocols.<n> GEO-Bench-2 addresses this with a comprehensive framework spanning classification, segmentation, regression, object detection, and instance segmentation.<n>Code, data, and leaderboard for GEO-Bench-2 are publicly released under a permissive license.
  arXiv  Detail & Related papers  (2025-11-19T17:45:02Z)
• Geospatial Diffusion for Land Cover Imperviousness Change Forecasting [0.0]
  We propose a new paradigm exploiting Generative AI (GenAI) for land cover change forecasting by framing LULC forecasting as a data problem conditioned on historical and auxiliary data-sources.<n>We train a diffusion model for decadal forecasting of imperviousness and compare its performance to a baseline that assumes no change at all.<n> evaluation across 12 metropolitan areas for a year held-out during training indicate that for average resolutions $geq 0.7times0.7km2$ our model yields MAE lower than such a baseline.
  arXiv  Detail & Related papers  (2025-08-14T13:45:10Z)
• A comparison of stretched-grid and limited-area modelling for data-driven regional weather forecasting [0.493599216374976]
  Limit-area model (LAM) and stretched-grid model (SGM) approaches have emerged for generating high-resolution regional forecasts.<n>This study aims to understand how the differences in model design impact relative performance and potential applications.
  arXiv  Detail & Related papers  (2025-07-24T12:54:08Z)
• WorldPrediction: A Benchmark for High-level World Modeling and Long-horizon Procedural Planning [52.36434784963598]
  We introduce WorldPrediction, a video-based benchmark for evaluating world modeling and procedural planning capabilities of different AI models.<n>We show that current frontier models barely achieve 57% accuracy on WorldPrediction-WM and 38% on WorldPrediction-PP whereas humans are able to solve both tasks perfectly.
  arXiv  Detail & Related papers  (2025-06-04T18:22:40Z)
• GAIA: A Foundation Model for Operational Atmospheric Dynamics [0.7454461126580372]
  GAIA is a novel model that combines masked autoencoders (MAE) and self-DIstillation with NO labels (DINO) for analyzing global atmospheric patterns in satellite imagery.<n>By integrating these complementary self-supervised learning approaches, our model simultaneously captures both local features and global dependencies.
  arXiv  Detail & Related papers  (2025-05-15T05:07:09Z)
• Towards an end-to-end artificial intelligence driven global weather forecasting system [57.5191940978886]
  We present an AI-based data assimilation model, i.e., Adas, for global weather variables. We demonstrate that Adas can assimilate global observations to produce high-quality analysis, enabling the system operate stably for long term. We are the first to apply the methods to real-world scenarios, which is more challenging and has considerable practical application potential.
  arXiv  Detail & Related papers  (2023-12-18T09:05:28Z)
• Residual Corrective Diffusion Modeling for Km-scale Atmospheric Downscaling [58.456404022536425]
  State of the art for physical hazard prediction from weather and climate requires expensive km-scale numerical simulations driven by coarser resolution global inputs. Here, a generative diffusion architecture is explored for downscaling such global inputs to km-scale, as a cost-effective machine learning alternative. The model is trained to predict 2km data from a regional weather model over Taiwan, conditioned on a 25km global reanalysis.
  arXiv  Detail & Related papers  (2023-09-24T19:57:22Z)
• Long-term drought prediction using deep neural networks based on geospatial weather data [75.38539438000072]
  High-quality drought forecasting up to a year in advance is critical for agriculture planning and insurance. We tackle drought data by introducing an end-to-end approach that adopts a systematic end-to-end approach. Key findings are the exceptional performance of a Transformer model, EarthFormer, in making accurate short-term (up to six months) forecasts.
  arXiv  Detail & Related papers  (2023-09-12T13:28:06Z)
• An Effective Meaningful Way to Evaluate Survival Models [34.21432603301076]
  In practice, the test set includes (right) censored individuals, meaning we do not know when a censored individual actually experienced the event. We introduce a novel and effective approach for generating realistic semi-synthetic survival datasets. Our proposed metric is able to rank models accurately based on their performance, and often closely matches the true MAE.
  arXiv  Detail & Related papers  (2023-06-01T23:22:46Z)
• Predictive World Models from Real-World Partial Observations [66.80340484148931]
  We present a framework for learning a probabilistic predictive world model for real-world road environments. While prior methods require complete states as ground truth for learning, we present a novel sequential training method to allow HVAEs to learn to predict complete states from partially observed states only.
  arXiv  Detail & Related papers  (2023-01-12T02:07:26Z)
• Pangu-Weather: A 3D High-Resolution Model for Fast and Accurate Global Weather Forecast [91.9372563527801]
  We present Pangu-Weather, a deep learning based system for fast and accurate global weather forecast. For the first time, an AI-based method outperforms state-of-the-art numerical weather prediction (NWP) methods in terms of accuracy. Pangu-Weather supports a wide range of downstream forecast scenarios, including extreme weather forecast and large-member ensemble forecast in real-time.
  arXiv  Detail & Related papers  (2022-11-03T17:19:43Z)
• A Meta-Analysis of Solar Forecasting Based on Skill Score [0.0]
  We conduct the first comprehensive meta-analysis of deterministic solar forecasting based on skill score. A database of 4,687 points was built and analyzed. More improvement is observed for intra-hour and intra-day than day-ahead forecasts.
  arXiv  Detail & Related papers  (2022-08-22T18:37:01Z)
• Leveraging Unlabeled Data to Predict Out-of-Distribution Performance [63.740181251997306]
  Real-world machine learning deployments are characterized by mismatches between the source (training) and target (test) distributions. In this work, we investigate methods for predicting the target domain accuracy using only labeled source data and unlabeled target data. We propose Average Thresholded Confidence (ATC), a practical method that learns a threshold on the model's confidence, predicting accuracy as the fraction of unlabeled examples.
  arXiv  Detail & Related papers  (2022-01-11T23:01:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.