An Ensemble Score Filter for Tracking High-Dimensional Nonlinear Dynamical Systems

• URL: http://arxiv.org/abs/2309.00983v2
• Date: Tue, 13 Aug 2024 14:48:39 GMT
• Title: An Ensemble Score Filter for Tracking High-Dimensional Nonlinear Dynamical Systems
• Authors: Feng Bao, Zezhong Zhang, Guannan Zhang,
• Abstract summary: We propose an ensemble score filter (EnSF) for solving high-dimensional nonlinear filtering problems. Unlike existing diffusion models that train neural networks to approximate the score function, we develop a training-free score estimation. EnSF provides surprising performance, compared with the state-of-the-art Local Ensemble Transform Kalman Filter method.
• Score: 10.997994515823798
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We propose an ensemble score filter (EnSF) for solving high-dimensional nonlinear filtering problems with superior accuracy. A major drawback of existing filtering methods, e.g., particle filters or ensemble Kalman filters, is the low accuracy in handling high-dimensional and highly nonlinear problems. EnSF attacks this challenge by exploiting the score-based diffusion model, defined in a pseudo-temporal domain, to characterizing the evolution of the filtering density. EnSF stores the information of the recursively updated filtering density function in the score function, instead of storing the information in a set of finite Monte Carlo samples (used in particle filters and ensemble Kalman filters). Unlike existing diffusion models that train neural networks to approximate the score function, we develop a training-free score estimation that uses a mini-batch-based Monte Carlo estimator to directly approximate the score function at any pseudo-spatial-temporal location, which provides sufficient accuracy in solving high-dimensional nonlinear problems as well as saves a tremendous amount of time spent on training neural networks. High-dimensional Lorenz-96 systems are used to demonstrate the performance of our method. EnSF provides surprising performance, compared with the state-of-the-art Local Ensemble Transform Kalman Filter method, in reliably and efficiently tracking extremely high-dimensional Lorenz systems (up to 1,000,000 dimensions) with highly nonlinear observation processes.

Related papers

• The Deep Latent Space Particle Filter for Real-Time Data Assimilation with Uncertainty Quantification [0.669618059970013]
  We present a novel particle filter methodology, the Deep Latent Space Particle filter or D-LSPF, that uses neural network-based surrogate models to overcome this computational challenge. The D-LSPF runs orders of magnitude faster than a high-fidelity particle filter and 3-5 times faster than alternative methods while being up to an order of magnitude more accurate.
  arXiv  Detail & Related papers  (2024-06-04T10:59:54Z)
• Outlier-robust Kalman Filtering through Generalised Bayes [45.51425214486509]
  We derive a novel, provably robust, and closed-form Bayesian update rule for online filtering in state-space models. Our method matches or outperforms other robust filtering methods at a much lower computational cost.
  arXiv  Detail & Related papers  (2024-05-09T09:40:56Z)
• Closed-form Filtering for Non-linear Systems [83.91296397912218]
  We propose a new class of filters based on Gaussian PSD Models, which offer several advantages in terms of density approximation and computational efficiency. We show that filtering can be efficiently performed in closed form when transitions and observations are Gaussian PSD Models. Our proposed estimator enjoys strong theoretical guarantees, with estimation error that depends on the quality of the approximation and is adaptive to the regularity of the transition probabilities.
  arXiv  Detail & Related papers  (2024-02-15T08:51:49Z)
• Nonlinear Filtering with Brenier Optimal Transport Maps [4.745059103971596]
  This paper is concerned with the problem of nonlinear filtering, i.e., computing the conditional distribution of the state of a dynamical system. Conventional sequential importance resampling (SIR) particle filters suffer from fundamental limitations, in scenarios involving degenerate likelihoods or high-dimensional states. In this paper, we explore an alternative method, which is based on estimating the Brenier optimal transport (OT) map from the current prior distribution of the state to the posterior distribution at the next time step.
  arXiv  Detail & Related papers  (2023-10-21T01:34:30Z)
• Low-rank extended Kalman filtering for online learning of neural networks from streaming data [71.97861600347959]
  We propose an efficient online approximate Bayesian inference algorithm for estimating the parameters of a nonlinear function from a potentially non-stationary data stream. The method is based on the extended Kalman filter (EKF), but uses a novel low-rank plus diagonal decomposition of the posterior matrix. In contrast to methods based on variational inference, our method is fully deterministic, and does not require step-size tuning.
  arXiv  Detail & Related papers  (2023-05-31T03:48:49Z)
• Computational Doob's h-transforms for Online Filtering of Discretely Observed Diffusions [65.74069050283998]
  We propose a computational framework to approximate Doob's $h$-transforms. The proposed approach can be orders of magnitude more efficient than state-of-the-art particle filters.
  arXiv  Detail & Related papers  (2022-06-07T15:03:05Z)
• Deep Learning for the Benes Filter [91.3755431537592]
  We present a new numerical method based on the mesh-free neural network representation of the density of the solution of the Benes model. We discuss the role of nonlinearity in the filtering model equations for the choice of the domain of the neural network.
  arXiv  Detail & Related papers  (2022-03-09T14:08:38Z)
• Batch Normalization Tells You Which Filter is Important [49.903610684578716]
  We propose a simple yet effective filter pruning method by evaluating the importance of each filter based on the BN parameters of pre-trained CNNs. The experimental results on CIFAR-10 and ImageNet demonstrate that the proposed method can achieve outstanding performance.
  arXiv  Detail & Related papers  (2021-12-02T12:04:59Z)
• KalmanNet: Neural Network Aided Kalman Filtering for Partially Known Dynamics [84.18625250574853]
  We present KalmanNet, a real-time state estimator that learns from data to carry out Kalman filtering under non-linear dynamics. We numerically demonstrate that KalmanNet overcomes nonlinearities and model mismatch, outperforming classic filtering methods.
  arXiv  Detail & Related papers  (2021-07-21T12:26:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.