Tools for Extracting Spatio-Temporal Patterns in Meteorological Image Sequences: From Feature Engineering to Attention-Based Neural Networks

• URL: http://arxiv.org/abs/2210.12310v2
• Date: Tue, 25 Oct 2022 03:59:28 GMT
• Title: Tools for Extracting Spatio-Temporal Patterns in Meteorological Image Sequences: From Feature Engineering to Attention-Based Neural Networks
• Authors: Akansha Singh Bansal, Yoonjin Lee, Kyle Hilburn and Imme Ebert-Uphoff
• Abstract summary: We review different concepts and techniques that are useful to extract context from image sequences. We first motivate the need for these approaches in meteorology using two applications, solar forecasting and detecting convection from satellite imagery. We provide an overview of many different concepts and techniques that are helpful for the interpretation of meteorological image sequences.
• Score: 5.566807756855081
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Atmospheric processes involve both space and time. This is why human analysis of atmospheric imagery can often extract more information from animated loops of image sequences than from individual images. Automating such an analysis requires the ability to identify spatio-temporal patterns in image sequences which is a very challenging task, because of the endless possibilities of patterns in both space and time. In this paper we review different concepts and techniques that are useful to extract spatio-temporal context specifically for meteorological applications. In this survey we first motivate the need for these approaches in meteorology using two applications, solar forecasting and detecting convection from satellite imagery. Then we provide an overview of many different concepts and techniques that are helpful for the interpretation of meteorological image sequences, such as (1) feature engineering methods to strengthen the desired signal in the input, using meteorological knowledge, classic image processing, harmonic analysis and topological data analysis (2) explain how different convolution filters (2D/3D/LSTM-convolution) can be utilized strategically in convolutional neural network architectures to find patterns in both space and time (3) discuss the powerful new concept of 'attention' in neural networks and the powerful abilities it brings to the interpretation of image sequences (4) briefly survey strategies from unsupervised, self-supervised and transfer learning to reduce the need for large labeled datasets. We hope that presenting an overview of these tools - many of which are underutilized - will help accelerate progress in this area.

Related papers

• Deep Multimodal Fusion for Semantic Segmentation of Remote Sensing Earth Observation Data [0.08192907805418582]
  This paper proposes a late fusion deep learning model (LF-DLM) for semantic segmentation. One branch integrates detailed textures from aerial imagery captured by UNetFormer with a Multi-Axis Vision Transformer (ViT) backbone. The other branch captures complex-temporal dynamics from the Sentinel-2 satellite imageMax time series using a U-ViNet with Temporal Attention (U-TAE)
  arXiv  Detail & Related papers  (2024-10-01T07:50:37Z)
• A Visual-Analytical Approach for Automatic Detection of Cyclonic Events in Satellite Observations [8.321173617981387]
  Estimating the location and intensity of tropical cyclones holds crucial significance for predicting catastrophic weather events. Current process for cyclone detection and intensity estimation involves physics-based simulation studies which are time-consuming. This research aims to focus specifically on cyclone detection, intensity estimation and related aspects using only image input and data-driven approaches.
  arXiv  Detail & Related papers  (2024-09-25T14:52:04Z)
• HVDistill: Transferring Knowledge from Images to Point Clouds via Unsupervised Hybrid-View Distillation [106.09886920774002]
  We present a hybrid-view-based knowledge distillation framework, termed HVDistill, to guide the feature learning of a point cloud neural network. Our method achieves consistent improvements over the baseline trained from scratch and significantly out- performs the existing schemes.
  arXiv  Detail & Related papers  (2024-03-18T14:18:08Z)
• BenchCloudVision: A Benchmark Analysis of Deep Learning Approaches for Cloud Detection and Segmentation in Remote Sensing Imagery [0.0]
  This paper examines seven cutting-edge semantic segmentation and detection algorithms applied to clouds identification. To increase the model's adaptability, critical elements including the type of imagery and the amount of spectral bands used during training are analyzed. Research tries to produce machine learning algorithms that can perform cloud segmentation using only a few spectral bands.
  arXiv  Detail & Related papers  (2024-02-21T16:32:43Z)
• Exploring the Application of Large-scale Pre-trained Models on Adverse Weather Removal [97.53040662243768]
  We propose a CLIP embedding module to make the network handle different weather conditions adaptively. This module integrates the sample specific weather prior extracted by CLIP image encoder together with the distribution specific information learned by a set of parameters.
  arXiv  Detail & Related papers  (2023-06-15T10:06:13Z)
• FuTH-Net: Fusing Temporal Relations and Holistic Features for Aerial Video Classification [49.06447472006251]
  We propose a novel deep neural network, termed FuTH-Net, to model not only holistic features, but also temporal relations for aerial video classification. Our model is evaluated on two aerial video classification datasets, ERA and Drone-Action, and achieves the state-of-the-art results.
  arXiv  Detail & Related papers  (2022-09-22T21:15:58Z)
• Spatial-Temporal Correlation and Topology Learning for Person Re-Identification in Videos [78.45050529204701]
  We propose a novel framework to pursue discriminative and robust representation by modeling cross-scale spatial-temporal correlation. CTL utilizes a CNN backbone and a key-points estimator to extract semantic local features from human body. It explores a context-reinforced topology to construct multi-scale graphs by considering both global contextual information and physical connections of human body.
  arXiv  Detail & Related papers  (2021-04-15T14:32:12Z)
• On the Post-hoc Explainability of Deep Echo State Networks for Time Series Forecasting, Image and Video Classification [63.716247731036745]
  echo state networks have attracted many stares through time, mainly due to the simplicity and computational efficiency of their learning algorithm. This work addresses this issue by conducting an explainability study of Echo State Networks when applied to learning tasks with time series, image and video data. Specifically, the study proposes three different techniques capable of eliciting understandable information about the knowledge grasped by these recurrent models.
  arXiv  Detail & Related papers  (2021-02-17T08:56:33Z)
• Probabilistic Graph Attention Network with Conditional Kernels for Pixel-Wise Prediction [158.88345945211185]
  We present a novel approach that advances the state of the art on pixel-level prediction in a fundamental aspect, i.e. structured multi-scale features learning and fusion. We propose a probabilistic graph attention network structure based on a novel Attention-Gated Conditional Random Fields (AG-CRFs) model for learning and fusing multi-scale representations in a principled manner.
  arXiv  Detail & Related papers  (2021-01-08T04:14:29Z)
• Evaluation, Tuning and Interpretation of Neural Networks for Meteorological Applications [0.030458514384586396]
  Neural networks have opened up many new opportunities to utilize remotely sensed images in meteorology. Common applications include image classification, e.g., to determine whether an image contains a tropical cyclone, and image translation, e.g., to emulate radar imagery for satellites that only have passive channels. This article highlights several strategies and practical considerations for neural network development that have not yet received much attention in the meteorological community.
  arXiv  Detail & Related papers  (2020-05-06T20:46:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.