A competitive baseline for deep learning enhanced data assimilation using conditional Gaussian ensemble Kalman filtering

• URL: http://arxiv.org/abs/2409.14300v1
• Date: Sun, 22 Sep 2024 02:54:33 GMT
• Title: A competitive baseline for deep learning enhanced data assimilation using conditional Gaussian ensemble Kalman filtering
• Authors: Zachariah Malik, Romit Maulik,
• Abstract summary: We study two non-linear extensions of the vanilla EnKF, dubbed the conditional-Gaussian EnKF (CG-EnKF) and the normal score EnKF (NS-EnKF) We compare these models against a state-of-the-art deep learning based particle filter called the score filter (SF) Our analysis also demonstrates that the CG-EnKF and NS-EnKF can handle highly non-Gaussian additive noise perturbations, with the latter typically outperforming the former.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Ensemble Kalman Filtering (EnKF) is a popular technique for data assimilation, with far ranging applications. However, the vanilla EnKF framework is not well-defined when perturbations are nonlinear. We study two non-linear extensions of the vanilla EnKF - dubbed the conditional-Gaussian EnKF (CG-EnKF) and the normal score EnKF (NS-EnKF) - which sidestep assumptions of linearity by constructing the Kalman gain matrix with the `conditional Gaussian' update formula in place of the traditional one. We then compare these models against a state-of-the-art deep learning based particle filter called the score filter (SF). This model uses an expensive score diffusion model for estimating densities and also requires a strong assumption on the perturbation operator for validity. In our comparison, we find that CG-EnKF and NS-EnKF dramatically outperform SF for a canonical problem in high-dimensional multiscale data assimilation given by the Lorenz-96 system. Our analysis also demonstrates that the CG-EnKF and NS-EnKF can handle highly non-Gaussian additive noise perturbations, with the latter typically outperforming the former.

Related papers

• Rectified Diffusion Guidance for Conditional Generation [62.00207951161297]
  We revisit the theory behind CFG and rigorously confirm that the improper configuration of the combination coefficients (i.e., the widely used summing-to-one version) brings about expectation shift of the generative distribution. We propose ReCFG with a relaxation on the guidance coefficients such that denoising with ReCFG strictly aligns with the diffusion theory. That way the rectified coefficients can be readily pre-computed via traversing the observed data, leaving the sampling speed barely affected.
  arXiv  Detail & Related papers  (2024-10-24T13:41:32Z)
• Closed-form Filtering for Non-linear Systems [83.91296397912218]
  We propose a new class of filters based on Gaussian PSD Models, which offer several advantages in terms of density approximation and computational efficiency. We show that filtering can be efficiently performed in closed form when transitions and observations are Gaussian PSD Models. Our proposed estimator enjoys strong theoretical guarantees, with estimation error that depends on the quality of the approximation and is adaptive to the regularity of the transition probabilities.
  arXiv  Detail & Related papers  (2024-02-15T08:51:49Z)
• Deep Kalman Filters Can Filter [9.131190818372474]
  Deep Kalman filters (DKFs) are a class of neural network models that generate Gaussian probability measures from sequential data. DKFs are inspired by the Kalman filter, but they lack concrete theoretical ties to the filtering problem. We show that continuous-time DKFs can implement the conditional law of a broad class of non-Markovian and conditionally Gaussian signal processes.
  arXiv  Detail & Related papers  (2023-10-30T14:58:12Z)
• Optimization or Architecture: How to Hack Kalman Filtering [52.640789351385266]
  In non-linear filtering, it is traditional to compare non-linear architectures such as neural networks to the standard linear Kalman Filter (KF) We argue that both should be optimized similarly, and to that end present the Optimized KF (OKF)
  arXiv  Detail & Related papers  (2023-10-01T14:00:18Z)
• Inverse Cubature and Quadrature Kalman filters [16.975704972827305]
  We develop inverse cubature KF (I-CKF), inverse quadrature KF (I-QKF), and inverse cubature-quadrature KF (I-CQKF) We derive the stability conditions for the proposed filters in the exponential-mean-squared-boundedness sense and prove the filters' consistency.
  arXiv  Detail & Related papers  (2023-03-18T03:48:39Z)
• Outlier-Insensitive Kalman Filtering Using NUV Priors [24.413595920205907]
  In practice, observations are corrupted by outliers, severely impairing the Kalman filter (KF)s performance. In this work, an outlier-insensitive KF is proposed, where is achieved by modeling each potential outlier as a normally distributed random variable with unknown variance (NUV) The NUVs variances are estimated online, using both expectation-maximization (EM) and alternating robustness (AM)
  arXiv  Detail & Related papers  (2022-10-12T11:00:13Z)
• Deep learning-enhanced ensemble-based data assimilation for high-dimensional nonlinear dynamical systems [0.0]
  Ensemble Kalman filter (EnKF) is a DA algorithm widely used in applications involving high-dimensional nonlinear dynamical systems. In this work, we propose hybrid ensemble Kalman filter (H-EnKF), which is applied to a two-layer quasi-geostrophic flow system as a test case.
  arXiv  Detail & Related papers  (2022-06-09T23:34:49Z)
• KalmanNet: Neural Network Aided Kalman Filtering for Partially Known Dynamics [84.18625250574853]
  We present KalmanNet, a real-time state estimator that learns from data to carry out Kalman filtering under non-linear dynamics. We numerically demonstrate that KalmanNet overcomes nonlinearities and model mismatch, outperforming classic filtering methods.
  arXiv  Detail & Related papers  (2021-07-21T12:26:46Z)
• Machine learning-based conditional mean filter: a generalization of the ensemble Kalman filter for nonlinear data assimilation [42.60602838972598]
  We propose a machine learning-based ensemble conditional mean filter (ML-EnCMF) for tracking possibly high-dimensional non-Gaussian state models with nonlinear dynamics based on sparse observations. The proposed filtering method is developed based on the conditional expectation and numerically implemented using machine learning (ML) techniques combined with the ensemble method.
  arXiv  Detail & Related papers  (2021-06-15T06:40:32Z)
• Autoregressive Score Matching [113.4502004812927]
  We propose autoregressive conditional score models (AR-CSM) where we parameterize the joint distribution in terms of the derivatives of univariable log-conditionals (scores) For AR-CSM models, this divergence between data and model distributions can be computed and optimized efficiently, requiring no expensive sampling or adversarial training. We show with extensive experimental results that it can be applied to density estimation on synthetic data, image generation, image denoising, and training latent variable models with implicit encoders.
  arXiv  Detail & Related papers  (2020-10-24T07:01:24Z)
• Gaussian MRF Covariance Modeling for Efficient Black-Box Adversarial Attacks [86.88061841975482]
  We study the problem of generating adversarial examples in a black-box setting, where we only have access to a zeroth order oracle. We use this setting to find fast one-step adversarial attacks, akin to a black-box version of the Fast Gradient Sign Method(FGSM) We show that the method uses fewer queries and achieves higher attack success rates than the current state of the art.
  arXiv  Detail & Related papers  (2020-10-08T18:36:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.