A deep mixture density network for outlier-corrected interpolation of crowd-sourced weather data

• URL: http://arxiv.org/abs/2201.10544v1
• Date: Tue, 25 Jan 2022 18:54:59 GMT
• Title: A deep mixture density network for outlier-corrected interpolation of crowd-sourced weather data
• Authors: Charlie Kirkwood, Theo Economou, Henry Odbert and Nicolas Pugeault
• Abstract summary: We present a deep learning approach for Bayesian-temporal modelling of environmental variables with automatic detection. For our example application, we use the Met Office's Weather Observation Website data, an archive of observations from around 1900 privately run and unofficial weather stations across the British Isles.
• Score: 3.1542695050861544
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: As the costs of sensors and associated IT infrastructure decreases - as exemplified by the Internet of Things - increasing volumes of observational data are becoming available for use by environmental scientists. However, as the number of available observation sites increases, so too does the opportunity for data quality issues to emerge, particularly given that many of these sensors do not have the benefit of official maintenance teams. To realise the value of crowd sourced 'Internet of Things' type observations for environmental modelling, we require approaches that can automate the detection of outliers during the data modelling process so that they do not contaminate the true distribution of the phenomena of interest. To this end, here we present a Bayesian deep learning approach for spatio-temporal modelling of environmental variables with automatic outlier detection. Our approach implements a Gaussian-uniform mixture density network whose dual purposes - modelling the phenomenon of interest, and learning to classify and ignore outliers - are achieved simultaneously, each by specifically designed branches of our neural network. For our example application, we use the Met Office's Weather Observation Website data, an archive of observations from around 1900 privately run and unofficial weather stations across the British Isles. Using data on surface air temperature, we demonstrate how our deep mixture model approach enables the modelling of a highly skilled spatio-temporal temperature distribution without contamination from spurious observations. We hope that adoption of our approach will help unlock the potential of incorporating a wider range of observation sources, including from crowd sourcing, into future environmental models.

Related papers

• PeFAD: A Parameter-Efficient Federated Framework for Time Series Anomaly Detection [51.20479454379662]
  We propose a. Federated Anomaly Detection framework named PeFAD with the increasing privacy concerns. We conduct extensive evaluations on four real datasets, where PeFAD outperforms existing state-of-the-art baselines by up to 28.74%.
  arXiv  Detail & Related papers  (2024-06-04T13:51:08Z)
• A Data-Driven Supervised Machine Learning Approach to Estimating Global Ambient Air Pollution Concentrations With Associated Prediction Intervals [0.0]
  We have developed a scalable, data-driven, supervised machine learning framework to impute missing temporal and spatial measurements. This model is designed to impute missing temporal and spatial measurements, thereby generating a comprehensive dataset for pollutants including NO$, O$_3$, PM$_10$, PM$_2.5$, and SO$. The model's performance across various geographical locations is examined, providing insights and recommendations for strategic placement of future monitoring stations.
  arXiv  Detail & Related papers  (2024-02-15T11:09:22Z)
• Observation-Guided Meteorological Field Downscaling at Station Scale: A Benchmark and a New Method [66.80344502790231]
  We extend meteorological downscaling to arbitrary scattered station scales and establish a new benchmark and dataset. Inspired by data assimilation techniques, we integrate observational data into the downscaling process, providing multi-scale observational priors. Our proposed method outperforms other specially designed baseline models on multiple surface variables.
  arXiv  Detail & Related papers  (2024-01-22T14:02:56Z)
• Mobile Internet Quality Estimation using Self-Tuning Kernel Regression [7.6449549886709764]
  We look into estimating mobile (cellular) internet quality at the scale of a state in the United States. Most of the samples are concentrated in limited areas, while very few are available in the rest. We propose a new adaptive kernel regression approach that employs self-tuning kernels to alleviate the adverse effects of data imbalance.
  arXiv  Detail & Related papers  (2023-11-04T21:09:46Z)
• SERT: A Transfomer Based Model for Spatio-Temporal Sensor Data with Missing Values for Environmental Monitoring [0.0]
  Data collected from sensors often contain missing values due to faulty equipment or maintenance issues. We propose two models that are capable of performing multivariate-temporal forecasting while handling missing data without need for imputation.
  arXiv  Detail & Related papers  (2023-06-05T17:06:23Z)
• Koopman-theoretic Approach for Identification of Exogenous Anomalies in Nonstationary Time-series Data [3.050919759387984]
  We build a general method for classifying anomalies in multi-dimensional time-series data. We demonstrate our proposed method on the important real-world task of global atmospheric pollution monitoring. The system successfully detects localized anomalies in air quality due to events such as COVID-19 lockdowns and wildfires.
  arXiv  Detail & Related papers  (2022-09-18T17:59:04Z)
• DAE : Discriminatory Auto-Encoder for multivariate time-series anomaly detection in air transportation [68.8204255655161]
  We propose a novel anomaly detection model called Discriminatory Auto-Encoder (DAE) It uses the baseline of a regular LSTM-based auto-encoder but with several decoders, each getting data of a specific flight phase. Results show that the DAE achieves better results in both accuracy and speed of detection.
  arXiv  Detail & Related papers  (2021-09-08T14:07:55Z)
• Lidar Light Scattering Augmentation (LISA): Physics-based Simulation of Adverse Weather Conditions for 3D Object Detection [60.89616629421904]
  Lidar-based object detectors are critical parts of the 3D perception pipeline in autonomous navigation systems such as self-driving cars. They are sensitive to adverse weather conditions such as rain, snow and fog due to reduced signal-to-noise ratio (SNR) and signal-to-background ratio (SBR)
  arXiv  Detail & Related papers  (2021-07-14T21:10:47Z)
• Deep Autoregressive Models with Spectral Attention [74.08846528440024]
  We propose a forecasting architecture that combines deep autoregressive models with a Spectral Attention (SA) module. By characterizing in the spectral domain the embedding of the time series as occurrences of a random process, our method can identify global trends and seasonality patterns. Two spectral attention models, global and local to the time series, integrate this information within the forecast and perform spectral filtering to remove time series's noise.
  arXiv  Detail & Related papers  (2021-07-13T11:08:47Z)
• Ill-posed Surface Emissivity Retrieval from Multi-Geometry HyperspectralImages using a Hybrid Deep Neural Network [0.0]
  Atmospheric correction is a fundamental task in remote sensing because observations are taken either of the atmosphere or looking through it. A geometry-dependent hybrid neural network is proposed for automatic atmospheric correction using multi-scan hyperspectral data. Results show that the proposed network has the capacity to accurately characterize the atmosphere and estimate target emissivity spectra with a Mean Absolute Error (MAE) under 0.02 for 29 different materials.
  arXiv  Detail & Related papers  (2021-07-09T18:59:58Z)
• Energy Aware Deep Reinforcement Learning Scheduling for Sensors Correlated in Time and Space [62.39318039798564]
  We propose a scheduling mechanism capable of taking advantage of correlated information. The proposed mechanism is capable of determining the frequency with which sensors should transmit their updates. We show that our solution can significantly extend the sensors' lifetime.
  arXiv  Detail & Related papers  (2020-11-19T09:53:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.