Deep learning-enhanced ensemble-based data assimilation for high-dimensional nonlinear dynamical systems

• URL: http://arxiv.org/abs/2206.04811v1
• Date: Thu, 9 Jun 2022 23:34:49 GMT
• Title: Deep learning-enhanced ensemble-based data assimilation for high-dimensional nonlinear dynamical systems
• Authors: Ashesh Chattopadhyay, Ebrahim Nabizadeh, Eviatar Bach, Pedram Hassanzadeh
• Abstract summary: Ensemble Kalman filter (EnKF) is a DA algorithm widely used in applications involving high-dimensional nonlinear dynamical systems. In this work, we propose hybrid ensemble Kalman filter (H-EnKF), which is applied to a two-layer quasi-geostrophic flow system as a test case.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Data assimilation (DA) is a key component of many forecasting models in science and engineering. DA allows one to estimate better initial conditions using an imperfect dynamical model of the system and noisy/sparse observations available from the system. Ensemble Kalman filter (EnKF) is a DA algorithm that is widely used in applications involving high-dimensional nonlinear dynamical systems. However, EnKF requires evolving large ensembles of forecasts using the dynamical model of the system. This often becomes computationally intractable, especially when the number of states of the system is very large, e.g., for weather prediction. With small ensembles, the estimated background error covariance matrix in the EnKF algorithm suffers from sampling error, leading to an erroneous estimate of the analysis state (initial condition for the next forecast cycle). In this work, we propose hybrid ensemble Kalman filter (H-EnKF), which is applied to a two-layer quasi-geostrophic flow system as a test case. This framework utilizes a pre-trained deep learning-based data-driven surrogate that inexpensively generates and evolves a large data-driven ensemble of the states of the system to accurately compute the background error covariance matrix with less sampling error. The H-EnKF framework estimates a better initial condition without the need for any ad-hoc localization strategies. H-EnKF can be extended to any ensemble-based DA algorithm, e.g., particle filters, which are currently difficult to use for high dimensional systems.

Related papers

• AI-Aided Kalman Filters [65.35350122917914]
  The Kalman filter (KF) and its variants are among the most celebrated algorithms in signal processing. Recent developments illustrate the possibility of fusing deep neural networks (DNNs) with classic Kalman-type filtering. This article provides a tutorial-style overview of design approaches for incorporating AI in aiding KF-type algorithms.
  arXiv  Detail & Related papers  (2024-10-16T06:47:53Z)
• Outlier-Insensitive Kalman Filtering Using NUV Priors [24.413595920205907]
  In practice, observations are corrupted by outliers, severely impairing the Kalman filter (KF)s performance. In this work, an outlier-insensitive KF is proposed, where is achieved by modeling each potential outlier as a normally distributed random variable with unknown variance (NUV) The NUVs variances are estimated online, using both expectation-maximization (EM) and alternating robustness (AM)
  arXiv  Detail & Related papers  (2022-10-12T11:00:13Z)
• $Φ$-DVAE: Physics-Informed Dynamical Variational Autoencoders for Unstructured Data Assimilation [3.2873782624127843]
  We develop a physics-informed dynamical variational autoencoder ($Phi$-DVAE) to embed diverse data streams into time-evolving physical systems. Our approach combines a standard, possibly nonlinear, filter for the latent state-space model and a VAE, to assimilate the unstructured data into the latent dynamical system. A variational Bayesian framework is used for the joint estimation of the encoding, latent states, and unknown system parameters.
  arXiv  Detail & Related papers  (2022-09-30T17:34:48Z)
• Generalised Latent Assimilation in Heterogeneous Reduced Spaces with Machine Learning Surrogate Models [10.410970649045943]
  We develop a system which combines reduced-order surrogate models with a novel data assimilation technique. Generalised Latent Assimilation can benefit both the efficiency provided by the reduced-order modelling and the accuracy of data assimilation.
  arXiv  Detail & Related papers  (2022-04-07T15:13:12Z)
• Scaling Structured Inference with Randomization [64.18063627155128]
  We propose a family of dynamic programming (RDP) randomized for scaling structured models to tens of thousands of latent states. Our method is widely applicable to classical DP-based inference. It is also compatible with automatic differentiation so can be integrated with neural networks seamlessly.
  arXiv  Detail & Related papers  (2021-12-07T11:26:41Z)
• Observation Error Covariance Specification in Dynamical Systems for Data assimilation using Recurrent Neural Networks [0.5330240017302621]
  We propose a data-driven approach based on long short term memory (LSTM) recurrent neural networks (RNN) The proposed approach does not require any knowledge or assumption about prior error distribution. We have compared the novel approach with two state-of-the-art covariance tuning algorithms, namely DI01 and D05.
  arXiv  Detail & Related papers  (2021-11-11T20:23:00Z)
• Large-Scale System Identification Using a Randomized SVD [4.567810220723372]
  We show that an approximate matrix factorization can replace the standard SVD in the realization algorithm. This is the only method capable of producing a model.
  arXiv  Detail & Related papers  (2021-09-06T19:25:15Z)
• KalmanNet: Neural Network Aided Kalman Filtering for Partially Known Dynamics [84.18625250574853]
  We present KalmanNet, a real-time state estimator that learns from data to carry out Kalman filtering under non-linear dynamics. We numerically demonstrate that KalmanNet overcomes nonlinearities and model mismatch, outperforming classic filtering methods.
  arXiv  Detail & Related papers  (2021-07-21T12:26:46Z)
• Using Data Assimilation to Train a Hybrid Forecast System that Combines Machine-Learning and Knowledge-Based Components [52.77024349608834]
  We consider the problem of data-assisted forecasting of chaotic dynamical systems when the available data is noisy partial measurements. We show that by using partial measurements of the state of the dynamical system, we can train a machine learning model to improve predictions made by an imperfect knowledge-based model.
  arXiv  Detail & Related papers  (2021-02-15T19:56:48Z)
• SUOD: Accelerating Large-Scale Unsupervised Heterogeneous Outlier Detection [63.253850875265115]
  Outlier detection (OD) is a key machine learning (ML) task for identifying abnormal objects from general samples. We propose a modular acceleration system, called SUOD, to address it.
  arXiv  Detail & Related papers  (2020-03-11T00:22:50Z)
• Simple and Effective Prevention of Mode Collapse in Deep One-Class Classification [93.2334223970488]
  We propose two regularizers to prevent hypersphere collapse in deep SVDD. The first regularizer is based on injecting random noise via the standard cross-entropy loss. The second regularizer penalizes the minibatch variance when it becomes too small.
  arXiv  Detail & Related papers  (2020-01-24T03:44:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.