Omni-Weather: Unified Multimodal Foundation Model for Weather Generation and Understanding

• URL: http://arxiv.org/abs/2512.21643v2
• Date: Mon, 29 Dec 2025 07:45:59 GMT
• Title: Omni-Weather: Unified Multimodal Foundation Model for Weather Generation and Understanding
• Authors: Zhiwang Zhou, Yuandong Pu, Xuming He, Yidi Liu, Yixin Chen, Junchao Gong, Xiang Zhuang, Wanghan Xu, Qinglong Cao, Shixiang Tang, Yihao Liu, Wenlong Zhang, Lei Bai,
• Abstract summary: We present Omni-Weather, the first multimodal foundation model that unifies weather generation and understanding within a single architecture.<n>Extensive experiments show Omni-Weather achieves state-of-the-art performance in both weather generation and understanding.<n>Our findings indicate that generative and understanding tasks in the weather domain can mutually enhance each other.
• Score: 45.208804604251405
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Weather modeling requires both accurate prediction and mechanistic interpretation, yet existing methods treat these goals in isolation, separating generation from understanding. To address this gap, we present Omni-Weather, the first multimodal foundation model that unifies weather generation and understanding within a single architecture. Omni-Weather integrates a radar encoder for weather generation tasks, followed by unified processing using a shared self-attention mechanism. Moreover, we construct a Chain-of-Thought dataset for causal reasoning in weather generation, enabling interpretable outputs and improved perceptual quality. Extensive experiments show Omni-Weather achieves state-of-the-art performance in both weather generation and understanding. Our findings further indicate that generative and understanding tasks in the weather domain can mutually enhance each other. Omni-Weather also demonstrates the feasibility and value of unifying weather generation and understanding.

Related papers

• Hierarchical AI-Meteorologist: LLM-Agent System for Multi-Scale and Explainable Weather Forecast Reporting [3.0354231393746685]
  Hierarchical AI-Meteorologist generates explainable weather reports using a hierarchical forecast reasoning and weather keyword generation.<n>Our framework performs multi-scale reasoning across hourly, 6-hour, and daily aggregations to capture both short-term dynamics and long-term trends.
  arXiv  Detail & Related papers  (2025-11-28T17:27:06Z)
• A Physics-guided Multimodal Transformer Path to Weather and Climate Sciences [59.05404971880922]
  Many problems in meteorology can now be addressed using AI models.<n>Data-driven algorithms have significantly improved accuracy compared to traditional methods.<n>We propose a new paradigm where observational data from different perspectives are treated as multimodal data and integrated via transformers.
  arXiv  Detail & Related papers  (2025-04-19T04:31:35Z)
• WeatherGFM: Learning A Weather Generalist Foundation Model via In-context Learning [69.82211470647349]
  We introduce the first generalist weather foundation model (WeatherGFM)<n>It addresses a wide spectrum of weather understanding tasks in a unified manner.<n>Our model can effectively handle up to ten weather understanding tasks, including weather forecasting, super-resolution, weather image translation, and post-processing.
  arXiv  Detail & Related papers  (2024-11-08T09:14:19Z)
• Advancing Meteorological Forecasting: AI-based Approach to Synoptic Weather Map Analysis [3.686808512438363]
  Our study proposes a novel preprocessing method and convolutional autoencoder model to improve the interpretation of synoptic weather maps. This model could recognize historical synoptic weather maps that nearly match current atmospheric conditions.
  arXiv  Detail & Related papers  (2024-11-08T07:46:50Z)
• WeatherQA: Can Multimodal Language Models Reason about Severe Weather? [45.43764278625153]
  Severe convective weather events, such as hail, tornadoes, and thunderstorms, often occur quickly yet cause significant damage, costing billions of dollars every year. This highlights the importance of forecasting severe weather threats hours in advance to better prepare meteorologists and residents in at-risk areas. We introduce WeatherQA, the first multimodal dataset designed for machines to reason about complex combinations of weather parameters and predict severe weather in real-world scenarios.
  arXiv  Detail & Related papers  (2024-06-17T05:23:18Z)
• Modeling Weather Uncertainty for Multi-weather Co-Presence Estimation [25.060597623607784]
  existing algorithms model weather condition as a discrete status and estimate it using multi-label classification. We consider the physical formulation of multi-weather conditions and model the impact of physical-related parameter on learning from the image appearance.
  arXiv  Detail & Related papers  (2024-03-29T10:05:29Z)
• Continual All-in-One Adverse Weather Removal with Knowledge Replay on a Unified Network Structure [92.8834309803903]
  In real-world applications, image degeneration caused by adverse weather is always complex and changes with different weather conditions from days and seasons. We develop a novel continual learning framework with effective knowledge replay (KR) on a unified network structure. It considers the characteristics of the image restoration task with multiple degenerations in continual learning, and the knowledge for different degenerations can be shared and accumulated.
  arXiv  Detail & Related papers  (2024-03-12T03:50:57Z)
• Towards an end-to-end artificial intelligence driven global weather forecasting system [57.5191940978886]
  We present an AI-based data assimilation model, i.e., Adas, for global weather variables. We demonstrate that Adas can assimilate global observations to produce high-quality analysis, enabling the system operate stably for long term. We are the first to apply the methods to real-world scenarios, which is more challenging and has considerable practical application potential.
  arXiv  Detail & Related papers  (2023-12-18T09:05:28Z)
• Learning Robust Precipitation Forecaster by Temporal Frame Interpolation [65.5045412005064]
  We develop a robust precipitation forecasting model that demonstrates resilience against spatial-temporal discrepancies. Our approach has led to significant improvements in forecasting precision, culminating in our model securing textit1st place in the transfer learning leaderboard of the textitWeather4cast'23 competition.
  arXiv  Detail & Related papers  (2023-11-30T08:22:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.