Finetuning a Weather Foundation Model with Lightweight Decoders for Unseen Physical Processes

• URL: http://arxiv.org/abs/2506.19088v1
• Date: Mon, 23 Jun 2025 20:03:53 GMT
• Title: Finetuning a Weather Foundation Model with Lightweight Decoders for Unseen Physical Processes
• Abstract summary: This study evaluates the performance of the state-of-the-art Aurora foundation model in predicting hydrological variables.<n>We introduce a lightweight approach using shallow decoders trained on the latent representations of the pretrained model to predict these new variables.
• Score: 25.060915780441086
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent advances in AI weather forecasting have led to the emergence of so-called "foundation models", typically defined by expensive pretraining and minimal fine-tuning for downstream tasks. However, in the natural sciences, a desirable foundation model should also encode meaningful statistical relationships between the underlying physical variables. This study evaluates the performance of the state-of-the-art Aurora foundation model in predicting hydrological variables, which were not considered during pretraining. We introduce a lightweight approach using shallow decoders trained on the latent representations of the pretrained model to predict these new variables. As a baseline, we compare this to fine-tuning the full model, which allows further optimization of the latent space while incorporating new variables into both inputs and outputs. The decoder-based approach requires 50% less training time and 35% less memory, while achieving strong accuracy across various hydrological variables and preserving desirable properties of the foundation model, such as autoregressive stability. Notably, decoder accuracy depends on the physical correlation between the new variables and those used during pretraining, indicating that Aurora's latent space captures meaningful physical relationships. In this sense, we argue that an important quality metric for foundation models in Earth sciences is their ability to be extended to new variables without a full fine-tuning. This provides a new perspective for making foundation models more accessible to communities with limited computational resources, while supporting broader adoption in Earth sciences.

Related papers

• On Neural Scaling Laws for Weather Emulation through Continual Training [76.70310203875866]
  We study neural scaling in Scientific Machine Learning, focusing on models for weather forecasting.<n>To analyze scaling behavior in as simple a setting as possible, we adopt a minimal, scalable, general-purpose Swin Transformer architecture.<n>We show that models trained in this minimalist way follow predictable scaling trends and even outperform standard cosine learning rate schedules.
  arXiv  Detail & Related papers  (2026-03-26T17:37:25Z)
• MAUNet-Light: A Concise MAUNet Architecture for Bias Correction and Downscaling of Precipitation Estimates [0.47248250311484113]
  This research aims to develop light-weight neural network architectures for both bias correction and downscaling of precipitation.<n>The proposed MAUNet-Light model is developed by transferring knowledge from the trained MAUNet.
  arXiv  Detail & Related papers  (2026-02-13T14:56:19Z)
• PhysE-Inv: A Physics-Encoded Inverse Modeling approach for Arctic Snow Depth Prediction [0.16206783799607727]
  We introduce PhysE-Inv, a novel framework that integrates a sophisticated sequential architecture, an LSTM-Decoder with Multi-head Attention and physics-guided contrastive learning, with physics-guided inference.<n> PhysE-Inv significantly improves prediction performance, reducing error by 20% while demonstrating superior physical consistency and resilience to data sparsity compared to empirical methods.<n>This approach pioneers a path for noise-tolerant, interpretable inverse modeling, with wide applicability in geospatial and cryospheric domains.
  arXiv  Detail & Related papers  (2026-01-23T00:43:51Z)
• Incorporating Multivariate Consistency in ML-Based Weather Forecasting with Latent-space Constraints [21.499040370144552]
  Data-driven machine learning (ML) models have recently shown promise in surpassing traditional physics-based approaches for weather forecasting.<n>Most ML-based forecast models treat reanalysis as the truth and are trained under variable-specific loss weighting.<n>Over long time horizons, the forecasts become blurry and physically unrealistic under rollout training.<n>We show that rollout training with latent-space constraints improves long-term forecast skill and better preserves fine-scale structures and physical realism.
  arXiv  Detail & Related papers  (2025-10-05T02:55:04Z)
• Scaling Laws for Emulation of Stellar Spectra [0.0]
  We provide training guidelines for scaling Transformer-based spectral emulators to achieve optimal performance.<n>Our results suggest that optimal computational resource allocation requires balanced scaling.<n>This study establishes a foundation for developing spectral foundational models with enhanced domain transfer capabilities.
  arXiv  Detail & Related papers  (2025-03-24T12:20:24Z)
• On conditional diffusion models for PDE simulations [53.01911265639582]
  We study score-based diffusion models for forecasting and assimilation of sparse observations. We propose an autoregressive sampling approach that significantly improves performance in forecasting. We also propose a new training strategy for conditional score-based models that achieves stable performance over a range of history lengths.
  arXiv  Detail & Related papers  (2024-10-21T18:31:04Z)
• Towards Physically Consistent Deep Learning For Climate Model Parameterizations [46.07009109585047]
  parameterizations are a major source of systematic errors and large uncertainties in climate projections. Deep learning (DL)-based parameterizations, trained on data from computationally expensive short, high-resolution simulations, have shown great promise for improving climate models. We propose an efficient supervised learning framework for DL-based parameterizations that leads to physically consistent models.
  arXiv  Detail & Related papers  (2024-06-06T10:02:49Z)
• Streamflow Prediction with Uncertainty Quantification for Water Management: A Constrained Reasoning and Learning Approach [27.984958596544278]
  This paper studies a constrained reasoning and learning (CRL) approach where physical laws represented as logical constraints are integrated as a layer in the deep neural network. To address small data setting, we develop a theoretically-grounded training approach to improve the generalization accuracy of deep models.
  arXiv  Detail & Related papers  (2024-05-31T18:53:53Z)
• When predict can also explain: few-shot prediction to select better neural latents [3.6218162133579703]
  Co-smoothing is used to estimate latent variables and predict observations along held-out channels.<n>In this study, we reveal the limitations of the co-smoothing prediction framework and propose a remedy.<n>We present a novel prediction metric designed to yield latent variables that more accurately reflect the ground truth.
  arXiv  Detail & Related papers  (2024-05-23T10:48:30Z)
• Learning Robust Precipitation Forecaster by Temporal Frame Interpolation [65.5045412005064]
  We develop a robust precipitation forecasting model that demonstrates resilience against spatial-temporal discrepancies. Our approach has led to significant improvements in forecasting precision, culminating in our model securing textit1st place in the transfer learning leaderboard of the textitWeather4cast'23 competition.
  arXiv  Detail & Related papers  (2023-11-30T08:22:08Z)
• Learning Generative Models for Lumped Rainfall-Runoff Modeling [3.69758875412828]
  This study presents a novel generative modeling approach to rainfall-runoff modeling, focusing on the synthesis of realistic daily catchment runoff time series. Unlike traditional process-based lumped hydrologic models, our approach uses a small number of latent variables to characterize runoff generation processes. In this study, we trained the generative models using neural networks on data from over 3,000 global catchments and achieved prediction accuracies comparable to current deep learning models.
  arXiv  Detail & Related papers  (2023-09-18T16:07:41Z)
• Stabilizing Machine Learning Prediction of Dynamics: Noise and Noise-inspired Regularization [58.720142291102135]
  Recent has shown that machine learning (ML) models can be trained to accurately forecast the dynamics of chaotic dynamical systems. In the absence of mitigating techniques, this technique can result in artificially rapid error growth, leading to inaccurate predictions and/or climate instability. We introduce Linearized Multi-Noise Training (LMNT), a regularization technique that deterministically approximates the effect of many small, independent noise realizations added to the model input during training.
  arXiv  Detail & Related papers  (2022-11-09T23:40:52Z)
• Human Trajectory Prediction via Neural Social Physics [63.62824628085961]
  Trajectory prediction has been widely pursued in many fields, and many model-based and model-free methods have been explored. We propose a new method combining both methodologies based on a new Neural Differential Equation model. Our new model (Neural Social Physics or NSP) is a deep neural network within which we use an explicit physics model with learnable parameters.
  arXiv  Detail & Related papers  (2022-07-21T12:11:18Z)
• Hyperparameter-free Continuous Learning for Domain Classification in Natural Language Understanding [60.226644697970116]
  Domain classification is the fundamental task in natural language understanding (NLU) Most existing continual learning approaches suffer from low accuracy and performance fluctuation. We propose a hyper parameter-free continual learning model for text data that can stably produce high performance under various environments.
  arXiv  Detail & Related papers  (2022-01-05T02:46:16Z)
• Physics-Integrated Variational Autoencoders for Robust and Interpretable Generative Modeling [86.9726984929758]
  We focus on the integration of incomplete physics models into deep generative models. We propose a VAE architecture in which a part of the latent space is grounded by physics. We demonstrate generative performance improvements over a set of synthetic and real-world datasets.
  arXiv  Detail & Related papers  (2021-02-25T20:28:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.