Exploring the design space of deep-learning-based weather forecasting systems

• URL: http://arxiv.org/abs/2410.07472v1
• Date: Wed, 9 Oct 2024 22:25:50 GMT
• Title: Exploring the design space of deep-learning-based weather forecasting systems
• Authors: Shoaib Ahmed Siddiqui, Jean Kossaifi, Boris Bonev, Christopher Choy, Jan Kautz, David Krueger, Kamyar Azizzadenesheli,
• Abstract summary: This paper systematically analyzes the impact of different design choices on deep-learning-based weather forecasting systems. We study fixed-grid architectures such as UNet, fully convolutional architectures, and transformer-based models. We propose a hybrid system that combines the strong performance of fixed-grid models with the flexibility of grid-invariant architectures.
• Score: 56.129148006412855
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Despite tremendous progress in developing deep-learning-based weather forecasting systems, their design space, including the impact of different design choices, is yet to be well understood. This paper aims to fill this knowledge gap by systematically analyzing these choices including architecture, problem formulation, pretraining scheme, use of image-based pretrained models, loss functions, noise injection, multi-step inputs, additional static masks, multi-step finetuning (including larger stride models), as well as training on a larger dataset. We study fixed-grid architectures such as UNet, fully convolutional architectures, and transformer-based models, along with grid-invariant architectures, including graph-based and operator-based models. Our results show that fixed-grid architectures outperform grid-invariant architectures, indicating a need for further architectural developments in grid-invariant models such as neural operators. We therefore propose a hybrid system that combines the strong performance of fixed-grid models with the flexibility of grid-invariant architectures. We further show that multi-step fine-tuning is essential for most deep-learning models to work well in practice, which has been a common practice in the past. Pretraining objectives degrade performance in comparison to supervised training, while image-based pretrained models provide useful inductive biases in some cases in comparison to training the model from scratch. Interestingly, we see a strong positive effect of using a larger dataset when training a smaller model as compared to training on a smaller dataset for longer. Larger models, on the other hand, primarily benefit from just an increase in the computational budget. We believe that these results will aid in the design of better weather forecasting systems in the future.

Related papers

• Transferable Post-training via Inverse Value Learning [83.75002867411263]
  We propose modeling changes at the logits level during post-training using a separate neural network (i.e., the value network) After training this network on a small base model using demonstrations, this network can be seamlessly integrated with other pre-trained models during inference. We demonstrate that the resulting value network has broad transferability across pre-trained models of different parameter sizes.
  arXiv  Detail & Related papers  (2024-10-28T13:48:43Z)
• Hierarchical and Decoupled BEV Perception Learning Framework for Autonomous Driving [52.808273563372126]
  This paper proposes a novel hierarchical BEV perception paradigm, aiming to provide a library of fundamental perception modules and user-friendly graphical interface. We conduct the Pretrain-Finetune strategy to effectively utilize large scale public datasets and streamline development processes. We also present a Multi-Module Learning (MML) approach, enhancing performance through synergistic and iterative training of multiple models.
  arXiv  Detail & Related papers  (2024-07-17T11:17:20Z)
• SortedNet: A Scalable and Generalized Framework for Training Modular Deep Neural Networks [30.069353400127046]
  We propose SortedNet to harness the inherent modularity of deep neural networks (DNNs) SortedNet enables the training of sub-models simultaneously along with the training of the main model. It is able to train 160 sub-models at once, achieving at least 96% of the original model's performance.
  arXiv  Detail & Related papers  (2023-09-01T05:12:25Z)
• Robust Graph Representation Learning via Predictive Coding [46.22695915912123]
  Predictive coding is a message-passing framework initially developed to model information processing in the brain. In this work, we build models that rely on the message-passing rule of predictive coding. We show that the proposed models are comparable to standard ones in terms of performance in both inductive and transductive tasks.
  arXiv  Detail & Related papers  (2022-12-09T03:58:22Z)
• RLFlow: Optimising Neural Network Subgraph Transformation with World Models [0.0]
  We propose a model-based agent which learns to optimise the architecture of neural networks by performing a sequence of subgraph transformations to reduce model runtime. We show our approach can match the performance of state of the art on common convolutional networks and outperform those by up to 5% on transformer-style architectures.
  arXiv  Detail & Related papers  (2022-05-03T11:52:54Z)
• An Intelligent End-to-End Neural Architecture Search Framework for Electricity Forecasting Model Development [4.940941112226529]
  We propose an intelligent automated architecture search (IAAS) framework for the development of time-series electricity forecasting models. The proposed framework contains three primary components, i.e., network function-preserving transformation operation, reinforcement learning (RL)-based network transformation control, and network screening. We demonstrate that the proposed IAAS framework significantly outperforms the ten existing models or methods in terms of forecasting accuracy and stability.
  arXiv  Detail & Related papers  (2022-03-25T10:36:27Z)
• Leveraging the structure of dynamical systems for data-driven modeling [111.45324708884813]
  We consider the impact of the training set and its structure on the quality of the long-term prediction. We show how an informed design of the training set, based on invariants of the system and the structure of the underlying attractor, significantly improves the resulting models.
  arXiv  Detail & Related papers  (2021-12-15T20:09:20Z)
• AdaXpert: Adapting Neural Architecture for Growing Data [63.30393509048505]
  In real-world applications, data often come in a growing manner, where the data volume and the number of classes may increase dynamically. Given the increasing data volume or the number of classes, one has to instantaneously adjust the neural model capacity to obtain promising performance. Existing methods either ignore the growing nature of data or seek to independently search an optimal architecture for a given dataset.
  arXiv  Detail & Related papers  (2021-07-01T07:22:05Z)
• Top-KAST: Top-K Always Sparse Training [50.05611544535801]
  We propose Top-KAST, a method that preserves constant sparsity throughout training. We show that it performs comparably to or better than previous works when training models on the established ImageNet benchmark. In addition to our ImageNet results, we also demonstrate our approach in the domain of language modeling.
  arXiv  Detail & Related papers  (2021-06-07T11:13:05Z)
• The Untapped Potential of Off-the-Shelf Convolutional Neural Networks [29.205446247063673]
  We show that existing off-the-shelf models like ResNet-50 are capable of over 95% accuracy on ImageNet. This level of performance currently exceeds that of models with over 20x more parameters and significantly more complex training procedures.
  arXiv  Detail & Related papers  (2021-03-17T20:04:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.