On Neural Scaling Laws for Weather Emulation through Continual Training

• URL: http://arxiv.org/abs/2603.25687v1
• Date: Thu, 26 Mar 2026 17:37:25 GMT
• Title: On Neural Scaling Laws for Weather Emulation through Continual Training
• Abstract summary: We study neural scaling in Scientific Machine Learning, focusing on models for weather forecasting.<n>To analyze scaling behavior in as simple a setting as possible, we adopt a minimal, scalable, general-purpose Swin Transformer architecture.<n>We show that models trained in this minimalist way follow predictable scaling trends and even outperform standard cosine learning rate schedules.
• Score: 76.70310203875866
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Neural scaling laws, which in some domains can predict the performance of large neural networks as a function of model, data, and compute scale, are the cornerstone of building foundation models in Natural Language Processing and Computer Vision. We study neural scaling in Scientific Machine Learning, focusing on models for weather forecasting. To analyze scaling behavior in as simple a setting as possible, we adopt a minimal, scalable, general-purpose Swin Transformer architecture, and we use continual training with constant learning rates and periodic cooldowns as an efficient training strategy. We show that models trained in this minimalist way follow predictable scaling trends and even outperform standard cosine learning rate schedules. Cooldown phases can be re-purposed to improve downstream performance, e.g., enabling accurate multi-step rollouts over longer forecast horizons as well as sharper predictions through spectral loss adjustments. We also systematically explore a wide range of model and dataset sizes under various compute budgets to construct IsoFLOP curves, and we identify compute-optimal training regimes. Extrapolating these trends to larger scales highlights potential performance limits, demonstrating that neural scaling can serve as an important diagnostic for efficient resource allocation. We open-source our code for reproducibility.

Related papers

• On the Practice of Scaling Search Conversion Rate Prediction [45.350775275846125]
  Scaling a Search Conversion Rate (CVR) prediction model, especially in high-traffic environments, presents a challenge.<n>This paper details an effective approach for scaling modern search CVR prediction models.
  arXiv  Detail & Related papers  (2026-05-28T01:48:19Z)
• The Art of Scaling Reinforcement Learning Compute for LLMs [52.71086085139566]
  Reinforcement learning (RL) has become central to training large language models.<n>Despite rapidly rising compute budgets, there is no principled understanding of how to evaluate algorithmic improvements for scaling RL compute.<n>We present the first large-scale systematic study, amounting to more than 400,000 GPU-hours.
  arXiv  Detail & Related papers  (2025-10-15T17:43:03Z)
• Enhancing material behavior discovery using embedding-oriented Physically-Guided Neural Networks with Internal Variables [0.0]
  Physically Guided Neural Networks with Internal Variables are SciML tools that use only observable data for training and unravel internal state relations.<n>Despite their potential, these models face challenges in scalability when applied to high-dimensional data such as fine-grid spatial fields or time-evolving systems.<n>We propose some enhancements to the PGNNIV framework that address these scalability limitations through reduced-order modeling techniques.
  arXiv  Detail & Related papers  (2025-08-01T12:33:21Z)
• Scaling Collapse Reveals Universal Dynamics in Compute-Optimally Trained Neural Networks [59.552873049024775]
  We show that compute-optimally trained models exhibit a remarkably precise universality.<n>With learning rate decay, the collapse becomes so tight that differences in the normalized curves across models fall below the noise floor.<n>We explain these phenomena by connecting collapse to the power-law structure in typical neural scaling laws.
  arXiv  Detail & Related papers  (2025-07-02T20:03:34Z)
• Neural Conformal Control for Time Series Forecasting [54.96087475179419]
  We introduce a neural network conformal prediction method for time series that enhances adaptivity in non-stationary environments.<n>Our approach acts as a neural controller designed to achieve desired target coverage, leveraging auxiliary multi-view data with neural network encoders.<n>We empirically demonstrate significant improvements in coverage and probabilistic accuracy, and find that our method is the only one that combines good calibration with consistency in prediction intervals.
  arXiv  Detail & Related papers  (2024-12-24T03:56:25Z)
• How Feature Learning Can Improve Neural Scaling Laws [79.59705237659547]
  We develop a solvable model of neural scaling laws beyond the kernel limit.<n>We show how performance scales with model size, training time, and the total amount of available data.
  arXiv  Detail & Related papers  (2024-09-26T14:05:32Z)
• A Dynamical Model of Neural Scaling Laws [79.59705237659547]
  We analyze a random feature model trained with gradient descent as a solvable model of network training and generalization. Our theory shows how the gap between training and test loss can gradually build up over time due to repeated reuse of data.
  arXiv  Detail & Related papers  (2024-02-02T01:41:38Z)
• RLFlow: Optimising Neural Network Subgraph Transformation with World Models [0.0]
  We propose a model-based agent which learns to optimise the architecture of neural networks by performing a sequence of subgraph transformations to reduce model runtime. We show our approach can match the performance of state of the art on common convolutional networks and outperform those by up to 5% on transformer-style architectures.
  arXiv  Detail & Related papers  (2022-05-03T11:52:54Z)
• Interpretable AI-based Large-scale 3D Pathloss Prediction Model for enabling Emerging Self-Driving Networks [3.710841042000923]
  We propose a Machine Learning-based model that leverages novel key predictors for estimating pathloss. By quantitatively evaluating the ability of various ML algorithms in terms of predictive, generalization and computational performance, our results show that Light Gradient Boosting Machine (LightGBM) algorithm overall outperforms others.
  arXiv  Detail & Related papers  (2022-01-30T19:50:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.