The impact of internal variability on benchmarking deep learning climate emulators

• URL: http://arxiv.org/abs/2408.05288v2
• Date: Mon, 31 Mar 2025 16:06:28 GMT
• Title: The impact of internal variability on benchmarking deep learning climate emulators
• Authors: Björn Lütjens, Raffaele Ferrari, Duncan Watson-Parris, Noelle Selin,
• Abstract summary: Full-complexity Earth system models (ESMs) are computationally very expensive, limiting their use in exploring the climate outcomes of multiple emission pathways.<n>More efficient emulators that approximate ESMs can directly map emissions onto climate outcomes, and benchmarks are being used to evaluate their accuracy on standardized tasks and datasets.<n>We investigate a popular benchmark in datadriven climate emulation, ClimateBench, on which deep learning-based emulators are currently achieving the best performance.
• Score: 2.3342885570554652
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Full-complexity Earth system models (ESMs) are computationally very expensive, limiting their use in exploring the climate outcomes of multiple emission pathways. More efficient emulators that approximate ESMs can directly map emissions onto climate outcomes, and benchmarks are being used to evaluate their accuracy on standardized tasks and datasets. We investigate a popular benchmark in data-driven climate emulation, ClimateBench, on which deep learning-based emulators are currently achieving the best performance. We compare these deep learning emulators with a linear regression-based emulator, akin to pattern scaling, and show that it outperforms the incumbent 100M-parameter deep learning foundation model, ClimaX, on 3 out of 4 regionally-resolved climate variables, notably surface temperature and precipitation. While emulating surface temperature is expected to be predominantly linear, this result is surprising for emulating precipitation. Precipitation is a much more noisy variable, and we show that deep learning emulators can overfit to internal variability noise at low frequencies, degrading their performance in comparison to a linear emulator. We address the issue of overfitting by increasing the number of climate simulations per emission pathway (from 3 to 50) and updating the benchmark targets with the respective ensemble averages from the MPI-ESM1.2-LR model. Using the new targets, we show that linear pattern scaling continues to be more accurate on temperature, but can be outperformed by a deep learning-based technique for emulating precipitation. We publish our code and data at github.com/blutjens/climate-emulator.

Related papers

• Autonomous Vehicle Controllers From End-to-End Differentiable Simulation [60.05963742334746]
  We propose a differentiable simulator and design an analytic policy gradients (APG) approach to training AV controllers. Our proposed framework brings the differentiable simulator into an end-to-end training loop, where gradients of environment dynamics serve as a useful prior to help the agent learn a more grounded policy. We find significant improvements in performance and robustness to noise in the dynamics, as well as overall more intuitive human-like handling.
  arXiv  Detail & Related papers  (2024-09-12T11:50:06Z)
• Machine learning emulation of precipitation from km-scale regional climate simulations using a diffusion model [22.255982502297197]
  High-resolution climate simulations are valuable for understanding climate change impacts. CPMGEM skilfully emulates precipitation simulations by a 2.2km-resolution regional CPM at much lower cost. It takes inputs at the 60km resolution of the driving global climate model and downscales these to 8.8km, with daily-mean time resolution.
  arXiv  Detail & Related papers  (2024-07-19T09:42:20Z)
• Uncertainty-enabled machine learning for emulation of regional sea-level change caused by the Antarctic Ice Sheet [0.8130739369606821]
  We build neural-network emulators of sea-level change at 27 coastal locations. We show that the neural-network emulators have an accuracy that is competitive with baseline machine learning emulators.
  arXiv  Detail & Related papers  (2024-06-21T18:27:09Z)
• NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and Benchmarking [65.24988062003096]
  We present NAVSIM, a framework for benchmarking vision-based driving policies. Our simulation is non-reactive, i.e., the evaluated policy and environment do not influence each other. NAVSIM enabled a new competition held at CVPR 2024, where 143 teams submitted 463 entries, resulting in several new insights.
  arXiv  Detail & Related papers  (2024-06-21T17:59:02Z)
• EM-GANSim: Real-time and Accurate EM Simulation Using Conditional GANs for 3D Indoor Scenes [55.2480439325792]
  We present a novel machine-learning (ML) approach (EM-GANSim) for real-time electromagnetic (EM) propagation. In practice, it can compute the signal strength in a few milliseconds on any location in 3D indoor environments.
  arXiv  Detail & Related papers  (2024-05-27T17:19:02Z)
• Sim2Real for Environmental Neural Processes [20.850715955359593]
  We analyse 'Sim2Real': pre-training on reanalysis and fine-tuning on observational data. Sim2Real could enable more accurate models for weather prediction and climate monitoring.
  arXiv  Detail & Related papers  (2023-10-30T18:49:06Z)
• Finding the Perfect Fit: Applying Regression Models to ClimateBench v1.0 [0.0]
  ClimateBench is a benchmarking dataset for evaluating the performance of machine learning emulators designed for climate data. This study focuses on evaluating non-linear regression models using the aforementioned dataset.
  arXiv  Detail & Related papers  (2023-08-23T01:08:01Z)
• FaIRGP: A Bayesian Energy Balance Model for Surface Temperatures Emulation [13.745581787463962]
  We introduce FaIRGP, a data-driven emulator that satisfies the physical temperature response equations of an energy balance model. We show how FaIRGP can be used to obtain estimates of top-of-atmosphere radiative forcing. We hope that this work will contribute to widening the adoption of data-driven methods in climate emulation.
  arXiv  Detail & Related papers  (2023-07-14T08:43:36Z)
• ClimaX: A foundation model for weather and climate [51.208269971019504]
  ClimaX is a deep learning model for weather and climate science. It can be pre-trained with a self-supervised learning objective on climate datasets. It can be fine-tuned to address a breadth of climate and weather tasks.
  arXiv  Detail & Related papers  (2023-01-24T23:19:01Z)
• Learning Large-scale Subsurface Simulations with a Hybrid Graph Network Simulator [57.57321628587564]
  We introduce Hybrid Graph Network Simulator (HGNS) for learning reservoir simulations of 3D subsurface fluid flows. HGNS consists of a subsurface graph neural network (SGNN) to model the evolution of fluid flows, and a 3D-U-Net to model the evolution of pressure. Using an industry-standard subsurface flow dataset (SPE-10) with 1.1 million cells, we demonstrate that HGNS is able to reduce the inference time up to 18 times compared to standard subsurface simulators.
  arXiv  Detail & Related papers  (2022-06-15T17:29:57Z)
• Deep Learning Based Cloud Cover Parameterization for ICON [55.49957005291674]
  We train NN based cloud cover parameterizations with coarse-grained data based on realistic regional and global ICON simulations. Globally trained NNs can reproduce sub-grid scale cloud cover of the regional simulation. We identify an overemphasis on specific humidity and cloud ice as the reason why our column-based NN cannot perfectly generalize from the global to the regional coarse-grained data.
  arXiv  Detail & Related papers  (2021-12-21T16:10:45Z)
• ClimART: A Benchmark Dataset for Emulating Atmospheric Radiative Transfer in Weather and Climate Models [13.514499533538789]
  We build a large dataset, ClimART, with more than emph10 million samples from present, pre-industrial, and future climate conditions, based on the Canadian Earth System Model. ClimART poses several methodological challenges for the ML community, such as multiple out-of-distribution test sets, underlying domain physics, and a trade-off between accuracy and inference speed. We also present several novel baselines that indicate shortcomings of datasets and network architectures used in prior work.
  arXiv  Detail & Related papers  (2021-11-29T16:32:31Z)
• Lidar Light Scattering Augmentation (LISA): Physics-based Simulation of Adverse Weather Conditions for 3D Object Detection [60.89616629421904]
  Lidar-based object detectors are critical parts of the 3D perception pipeline in autonomous navigation systems such as self-driving cars. They are sensitive to adverse weather conditions such as rain, snow and fog due to reduced signal-to-noise ratio (SNR) and signal-to-background ratio (SBR)
  arXiv  Detail & Related papers  (2021-07-14T21:10:47Z)
• DriveGAN: Towards a Controllable High-Quality Neural Simulation [147.6822288981004]
  We introduce a novel high-quality neural simulator referred to as DriveGAN. DriveGAN achieves controllability by disentangling different components without supervision. We train DriveGAN on multiple datasets, including 160 hours of real-world driving data.
  arXiv  Detail & Related papers  (2021-04-30T15:30:05Z)
• DeepClimGAN: A High-Resolution Climate Data Generator [60.59639064716545]
  Earth system models (ESMs) are often used to generate future projections of climate change scenarios. As a compromise, emulators are substantially less expensive but may not have all of the complexity of an ESM. Here we demonstrate the use of a conditional generative adversarial network (GAN) to act as an ESM emulator.
  arXiv  Detail & Related papers  (2020-11-23T20:13:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.