Personalized Adapter for Large Meteorology Model on Devices: Towards Weather Foundation Models

• URL: http://arxiv.org/abs/2405.20348v1
• Date: Fri, 24 May 2024 15:25:09 GMT
• Title: Personalized Adapter for Large Meteorology Model on Devices: Towards Weather Foundation Models
• Authors: Shengchao Chen, Guodong Long, Jing Jiang, Chengqi Zhang,
• Abstract summary: LM-Weather is a generic approach to taming pre-trained language models (PLMs) We introduce a lightweight personalized adapter into PLMs and endow it with weather pattern awareness. Experiments show LM-Weather outperforms the state-of-the-art results by a large margin across various tasks.
• Score: 36.229082478423585
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper demonstrates that pre-trained language models (PLMs) are strong foundation models for on-device meteorological variables modeling. We present LM-Weather, a generic approach to taming PLMs, that have learned massive sequential knowledge from the universe of natural language databases, to acquire an immediate capability to obtain highly customized models for heterogeneous meteorological data on devices while keeping high efficiency. Concretely, we introduce a lightweight personalized adapter into PLMs and endows it with weather pattern awareness. During communication between clients and the server, low-rank-based transmission is performed to effectively fuse the global knowledge among devices while maintaining high communication efficiency and ensuring privacy. Experiments on real-wold dataset show that LM-Weather outperforms the state-of-the-art results by a large margin across various tasks (e.g., forecasting and imputation at different scales). We provide extensive and in-depth analyses experiments, which verify that LM-Weather can (1) indeed leverage sequential knowledge from natural language to accurately handle meteorological sequence, (2) allows each devices obtain highly customized models under significant heterogeneity, and (3) generalize under data-limited and out-of-distribution (OOD) scenarios.

Related papers

• A Scalable Real-Time Data Assimilation Framework for Predicting Turbulent Atmosphere Dynamics [8.012940782999975]
  We introduce a generic real-time data assimilation framework and demonstrate its end-to-end performance on the Frontier supercomputer. This framework comprises two primary modules: an ensemble score filter (EnSF) and a vision transformer-based surrogate. We demonstrate both the strong and weak scaling of our framework up to 1024 GPUs on the Exascale supercomputer, Frontier.
  arXiv  Detail & Related papers  (2024-07-16T20:44:09Z)
• Generalizing Weather Forecast to Fine-grained Temporal Scales via Physics-AI Hybrid Modeling [55.13352174687475]
  This paper proposes a physics-AI hybrid model (i.e., WeatherGFT) which Generalizes weather forecasts to Finer-grained Temporal scales. Specifically, we employ a carefully designed PDE kernel to simulate physical evolution on a small time scale. We introduce a lead time-aware training framework to promote the generalization of the model at different lead times.
  arXiv  Detail & Related papers  (2024-05-22T16:21:02Z)
• Towards an end-to-end artificial intelligence driven global weather forecasting system [57.5191940978886]
  We present an AI-based data assimilation model, i.e., Adas, for global weather variables. We demonstrate that Adas can assimilate global observations to produce high-quality analysis, enabling the system operate stably for long term. We are the first to apply the methods to real-world scenarios, which is more challenging and has considerable practical application potential.
  arXiv  Detail & Related papers  (2023-12-18T09:05:28Z)
• FengWu-4DVar: Coupling the Data-driven Weather Forecasting Model with 4D Variational Assimilation [67.20588721130623]
  We develop an AI-based cyclic weather forecasting system, FengWu-4DVar. FengWu-4DVar can incorporate observational data into the data-driven weather forecasting model. Experiments on the simulated observational dataset demonstrate that FengWu-4DVar is capable of generating reasonable analysis fields.
  arXiv  Detail & Related papers  (2023-12-16T02:07:56Z)
• A Distributed Approach to Meteorological Predictions: Addressing Data Imbalance in Precipitation Prediction Models through Federated Learning and GANs [0.0]
  classification of weather data involves categorizing meteorological phenomena into classes, thereby facilitating nuanced analyses and precise predictions. It's imperative that classification algorithms proficiently navigate challenges such as data imbalances. Data augmentation techniques can improve the model's accuracy in classifying rare but critical weather events.
  arXiv  Detail & Related papers  (2023-10-19T21:28:20Z)
• Federated Prompt Learning for Weather Foundation Models on Devices [37.88417074427373]
  On-device intelligence for weather forecasting uses local deep learning models to analyze weather patterns without centralized cloud computing. This paper propose Federated Prompt Learning for Weather Foundation Models on Devices (FedPoD) FedPoD enables devices to obtain highly customized models while maintaining communication efficiency.
  arXiv  Detail & Related papers  (2023-05-23T16:59:20Z)
• ClimaX: A foundation model for weather and climate [51.208269971019504]
  ClimaX is a deep learning model for weather and climate science. It can be pre-trained with a self-supervised learning objective on climate datasets. It can be fine-tuned to address a breadth of climate and weather tasks.
  arXiv  Detail & Related papers  (2023-01-24T23:19:01Z)
• Prompt Federated Learning for Weather Forecasting: Toward Foundation Models on Meteorological Data [37.549578998407675]
  To tackle the global climate challenge, it urgently needs to develop a collaborative platform for comprehensive weather forecasting on large-scale meteorological data. This paper develops a foundation model across regions of understanding complex meteorological data and providing weather forecasting. A novel prompt learning mechanism has been adopted to satisfy low-resourced sensors' communication and computational constraints.
  arXiv  Detail & Related papers  (2023-01-22T16:47:05Z)
• Large Language Models with Controllable Working Memory [64.71038763708161]
  Large language models (LLMs) have led to a series of breakthroughs in natural language processing (NLP) What further sets these models apart is the massive amounts of world knowledge they internalize during pretraining. How the model's world knowledge interacts with the factual information presented in the context remains under explored.
  arXiv  Detail & Related papers  (2022-11-09T18:58:29Z)
• Numerical Weather Forecasting using Convolutional-LSTM with Attention and Context Matcher Mechanisms [10.759556555869798]
  We introduce a novel deep learning architecture for forecasting high-resolution weather data. Our Weather Model achieves significant performance improvements compared to baseline deep learning models.
  arXiv  Detail & Related papers  (2021-02-01T08:30:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.