RealBench: Benchmarking Data-Driven Numerical Weather Forecasting Under Operational Conditions and Extreme Event Challenges

• URL: http://arxiv.org/abs/2605.24945v1
• Date: Sun, 24 May 2026 08:46:17 GMT
• Title: RealBench: Benchmarking Data-Driven Numerical Weather Forecasting Under Operational Conditions and Extreme Event Challenges
• Abstract summary: We introduce RealBench, a next-generation benchmark for AI weather forecasting that emphasizes realistic evaluation under operational conditions.<n>RealBench features a strictly out-of-distribution test set spanning 2025 to eliminate data leakage and capture recent atmospheric regimes.<n>It integrates multiple data sources, including low-latency operational analysis and a large-scale global in-situ observation dataset comprising over 10,000 stations.
• Score: 31.389267895745252
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Accurate evaluation of weather forecasting models is critical for their reliable deployment in real-world applications. However, existing benchmarks predominantly rely on reanalysis products such as ERA5, which are generated through delayed data assimilation and do not reflect the constraints of real-time operational forecasting, thereby resulting in a systematic mismatch between benchmark performance and real-world forecasting. In this work, we introduce RealBench, a next-generation benchmark for AI weather forecasting that emphasizes realistic evaluation under operational conditions. RealBench features a strictly out-of-distribution test set spanning 2025 to eliminate data leakage and capture recent atmospheric regimes. It integrates multiple data sources, including low-latency operational analysis and a large-scale global in-situ observation dataset comprising over 10,000 stations, enabling direct evaluation against real atmospheric measurements. Beyond standard global metrics, RealBench provides a comprehensive evaluation framework for high-impact extreme events, including heatwaves, cold surges, and tropical cyclones, using event-specific metrics that better reflect real-world forecasting priorities. The evaluation results reveal substantial discrepancies between reanalysis-based metrics and real-world performance, particularly concerning extreme events. By highlighting the limitations of existing benchmarks, this work establishes a more faithful and operationally relevant evaluation paradigm, providing a rigorous foundation for advancing next-generation AI weather forecasting systems. The benchmark implementation is available at: https://github.com/lixruize-del/NWP-Benchmark.

Related papers

• TSSM: Triaxial State Space Model for Global Station Weather Forecasting with Temporal-Variable-Historical Modeling [74.915718648935]
  Existing methods show limited accuracy gains and struggle with extreme events and error accumulation.<n>We propose a novel Triaxial State Space Model (TSSM) with a history-enhanced Temporal-VariableHistorical paradigm.<n>TSSM stacks historical samples into period-aligned batches, where forecasting is causally supported by historical and current observations.
  arXiv  Detail & Related papers  (2026-07-14T07:36:19Z)
• EO-WM: A Physically Informed World Model for Probabilistic Earth Observation Forecasting [50.31500063307306]
  We introduce EO-WM, a video diffusion transformer for multispectral Earth Observation forecasting.<n> EO-WM incorporates a physically informed conditioning framework that represents meteorological forcing through a climatological baseline, weather anomalies, and cumulative physical stress signals.<n>Experiments show that EO-WM reduces the error in predicted Normalized Difference Vegetation Index (NDVI) decline amplitude by a relative 5.63% and improves directional hit rate by a relative 7.80%.
  arXiv  Detail & Related papers  (2026-06-25T16:53:30Z)
• AirQualityBench: A Realistic Evaluation Benchmark for Global Air Quality Forecasting [25.500339307707225]
  We introduce StarbfAirQualityBench, a global multi-pollutant benchmark to evaluate forecasting models under realistic conditions.<n>The benchmark contains hourly observations from 3,720 monitoring stations over 2021--2025, covers six major pollutants, and preserves provider-native observation masks.<n>AirQualityBench shows that strong performance on sanitized datasets does not reliably transfer to global, fragmented monitoring streams.
  arXiv  Detail & Related papers  (2026-05-07T08:25:43Z)
• TFRBench: A Reasoning Benchmark for Evaluating Forecasting Systems [52.91956121737963]
  We introduce TFRBench, the first benchmark designed to evaluate the reasoning capabilities of forecasting systems.<n>We propose a systematic multi-agent framework that utilizes an iterative verification loop to synthesize numerically grounded reasoning traces.<n>Our evaluation confirms that this reasoning is causally effective; useful for evaluation; and prompting LLMs with our generated traces significantly improves forecasting accuracy.
  arXiv  Detail & Related papers  (2026-04-07T03:04:45Z)
• FuXiWeather2: Learning accurate atmospheric state estimation for operational global weather forecasting [15.10103712842682]
  We present FuXiWeather2, a unified end-to-end neural framework for assimilation and forecasting.<n>FuXiWeather2 generates high-resolution ($0.25circ$) global analysis fields and 10-day forecasts within minutes.<n>These high-quality analysis fields drive deterministic forecasts that exceed the skill of the HRES system in 91% of evaluated metrics.
  arXiv  Detail & Related papers  (2026-03-16T14:36:47Z)
• Echo State Networks for Time Series Forecasting: Hyperparameter Sweep and Benchmarking [51.56484100374058]
  We evaluate whether a fully automatic, purely feedback-driven ESN can serve as a competitive alternative to widely used statistical forecasting methods.<n>Forecast accuracy is measured using MASE and sMAPE and benchmarked against simple benchmarks like drift and seasonal naive and statistical models.
  arXiv  Detail & Related papers  (2026-02-03T16:01:22Z)
• WeatherReal: A Benchmark Based on In-Situ Observations for Evaluating Weather Models [11.016845506758841]
  We introduce WeatherReal, a novel benchmark dataset for weather forecasting derived from global near-surface in-situ observations. This paper details the sources and processing methodologies underlying the dataset, and illustrates the advantage of in-situ observations in capturing hyper-local and extreme weather. Our work aims to advance the AI-based weather forecasting research towards a more application-focused and operation-ready approach.
  arXiv  Detail & Related papers  (2024-09-14T08:53:46Z)
• How far are today's time-series models from real-world weather forecasting applications? [22.68937280154092]
  WEATHER-5K is a comprehensive collection of observational weather data that better reflects real-world scenarios. It enables a better training of models and a more accurate assessment of the real-world forecasting capabilities of TSF models. We provide researchers with a clear assessment of the gap between academic TSF models and real-world weather forecasting applications.
  arXiv  Detail & Related papers  (2024-06-20T15:18:52Z)
• ExtremeCast: Boosting Extreme Value Prediction for Global Weather Forecast [57.6987191099507]
  We introduce Exloss, a novel loss function that performs asymmetric optimization and highlights extreme values to obtain accurate extreme weather forecast. We also introduce ExBooster, which captures the uncertainty in prediction outcomes by employing multiple random samples. Our solution can achieve state-of-the-art performance in extreme weather prediction, while maintaining the overall forecast accuracy comparable to the top medium-range forecast models.
  arXiv  Detail & Related papers  (2024-02-02T10:34:13Z)
• FengWu-GHR: Learning the Kilometer-scale Medium-range Global Weather Forecasting [56.73502043159699]
  This work presents FengWu-GHR, the first data-driven global weather forecasting model running at the 0.09$circ$ horizontal resolution. It introduces a novel approach that opens the door for operating ML-based high-resolution forecasts by inheriting prior knowledge from a low-resolution model. The hindcast of weather prediction in 2022 indicates that FengWu-GHR is superior to the IFS-HRES.
  arXiv  Detail & Related papers  (2024-01-28T13:23:25Z)
• Kunyu: A High-Performing Global Weather Model Beyond Regression Losses [0.0]
  I present Kunyu, a global data-driven weather forecasting model which delivers accurate predictions across a comprehensive array of atmospheric variables at 0.35deg resolution. With both regression and adversarial losses integrated in its training framework, Kunyu generates forecasts with enhanced clarity and realism.
  arXiv  Detail & Related papers  (2023-12-04T17:30:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.