Vision-Language Models Meet Meteorology: Developing Models for Extreme Weather Events Detection with Heatmaps
- URL: http://arxiv.org/abs/2406.09838v1
- Date: Fri, 14 Jun 2024 08:46:44 GMT
- Title: Vision-Language Models Meet Meteorology: Developing Models for Extreme Weather Events Detection with Heatmaps
- Authors: Jian Chen, Peilin Zhou, Yining Hua, Dading Chong, Meng Cao, Yaowei Li, Zixuan Yuan, Bing Zhu, Junwei Liang,
- Abstract summary: Real-time detection and prediction of extreme weather protect human lives and infrastructure.
Traditional methods rely on numerical threshold setting and manual interpretation of weather heatmaps with Geographic Information Systems (GIS)
Our research redefines Extreme Weather Events Detection (EWED) by framing it as a Visual Question Answering (VQA) problem.
- Score: 30.771706309741656
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Real-time detection and prediction of extreme weather protect human lives and infrastructure. Traditional methods rely on numerical threshold setting and manual interpretation of weather heatmaps with Geographic Information Systems (GIS), which can be slow and error-prone. Our research redefines Extreme Weather Events Detection (EWED) by framing it as a Visual Question Answering (VQA) problem, thereby introducing a more precise and automated solution. Leveraging Vision-Language Models (VLM) to simultaneously process visual and textual data, we offer an effective aid to enhance the analysis process of weather heatmaps. Our initial assessment of general-purpose VLMs (e.g., GPT-4-Vision) on EWED revealed poor performance, characterized by low accuracy and frequent hallucinations due to inadequate color differentiation and insufficient meteorological knowledge. To address these challenges, we introduce ClimateIQA, the first meteorological VQA dataset, which includes 8,760 wind gust heatmaps and 254,040 question-answer pairs covering four question types, both generated from the latest climate reanalysis data. We also propose Sparse Position and Outline Tracking (SPOT), an innovative technique that leverages OpenCV and K-Means clustering to capture and depict color contours in heatmaps, providing ClimateIQA with more accurate color spatial location information. Finally, we present Climate-Zoo, the first meteorological VLM collection, which adapts VLMs to meteorological applications using the ClimateIQA dataset. Experiment results demonstrate that models from Climate-Zoo substantially outperform state-of-the-art general VLMs, achieving an accuracy increase from 0% to over 90% in EWED verification. The datasets and models in this study are publicly available for future climate science research: https://github.com/AlexJJJChen/Climate-Zoo.
Related papers
- ACE Metric: Advection and Convection Evaluation for Accurate Weather Forecasting [7.016835396874093]
We propose the advection and convection Error (ACE) metric to assess how well models predict advection and convection.
We have validated the ACE evaluation metric on the WeatherBench2 and MovingMNIST datasets.
arXiv Detail & Related papers (2024-06-07T06:49:59Z) - Advancing Data-driven Weather Forecasting: Time-Sliding Data
Augmentation of ERA5 [3.3748750222488657]
We introduce a novel strategy that deviates from the common dependence on high-resolution data.
This paper improves on conventional approaches by adding more variables and a novel approach to data augmentation and processing.
Our findings reveal that despite the lower resolution, the proposed approach demonstrates considerable accuracy in predicting atmospheric conditions.
arXiv Detail & Related papers (2024-02-13T03:01:22Z) - FengWu-GHR: Learning the Kilometer-scale Medium-range Global Weather
Forecasting [56.73502043159699]
This work presents FengWu-GHR, the first data-driven global weather forecasting model running at the 0.09$circ$ horizontal resolution.
It introduces a novel approach that opens the door for operating ML-based high-resolution forecasts by inheriting prior knowledge from a low-resolution model.
The hindcast of weather prediction in 2022 indicates that FengWu-GHR is superior to the IFS-HRES.
arXiv Detail & Related papers (2024-01-28T13:23:25Z) - Observation-Guided Meteorological Field Downscaling at Station Scale: A
Benchmark and a New Method [66.80344502790231]
We extend meteorological downscaling to arbitrary scattered station scales and establish a new benchmark and dataset.
Inspired by data assimilation techniques, we integrate observational data into the downscaling process, providing multi-scale observational priors.
Our proposed method outperforms other specially designed baseline models on multiple surface variables.
arXiv Detail & Related papers (2024-01-22T14:02:56Z) - Self Supervised Vision for Climate Downscaling [16.407155686685666]
Future projections for climate change research are based on Earth System Models (ESMs), the computer models that simulate the Earth's climate system.
ESMs provide a framework to integrate various physical systems, but their output is bound by the enormous computational resources required for running and archiving higher-resolution simulations.
In this work, we present a deep-learning model for downscaling ESM simulation data that does not require high-resolution ground truth data for model optimization.
arXiv Detail & Related papers (2024-01-09T10:20:49Z) - Towards an end-to-end artificial intelligence driven global weather forecasting system [57.5191940978886]
We present an AI-based data assimilation model, i.e., Adas, for global weather variables.
We demonstrate that Adas can assimilate global observations to produce high-quality analysis, enabling the system operate stably for long term.
We are the first to apply the methods to real-world scenarios, which is more challenging and has considerable practical application potential.
arXiv Detail & Related papers (2023-12-18T09:05:28Z) - ClimaX: A foundation model for weather and climate [51.208269971019504]
ClimaX is a deep learning model for weather and climate science.
It can be pre-trained with a self-supervised learning objective on climate datasets.
It can be fine-tuned to address a breadth of climate and weather tasks.
arXiv Detail & Related papers (2023-01-24T23:19:01Z) - GraphCast: Learning skillful medium-range global weather forecasting [107.40054095223779]
We introduce a machine learning-based method called "GraphCast", which can be trained directly from reanalysis data.
It predicts hundreds of weather variables, over 10 days at 0.25 degree resolution globally, in under one minute.
We show that GraphCast significantly outperforms the most accurate operational deterministic systems on 90% of 1380 verification targets.
arXiv Detail & Related papers (2022-12-24T18:15:39Z) - Pangu-Weather: A 3D High-Resolution Model for Fast and Accurate Global
Weather Forecast [91.9372563527801]
We present Pangu-Weather, a deep learning based system for fast and accurate global weather forecast.
For the first time, an AI-based method outperforms state-of-the-art numerical weather prediction (NWP) methods in terms of accuracy.
Pangu-Weather supports a wide range of downstream forecast scenarios, including extreme weather forecast and large-member ensemble forecast in real-time.
arXiv Detail & Related papers (2022-11-03T17:19:43Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.