The Deep Latent Space Particle Filter for Real-Time Data Assimilation with Uncertainty Quantification

• URL: http://arxiv.org/abs/2406.02204v1
• Date: Tue, 4 Jun 2024 10:59:54 GMT
• Title: The Deep Latent Space Particle Filter for Real-Time Data Assimilation with Uncertainty Quantification
• Authors: Nikolaj T. Mücke, Sander M. Bohté, Cornelis W. Oosterlee,
• Abstract summary: We present a novel particle filter methodology, the Deep Latent Space Particle filter or D-LSPF, that uses neural network-based surrogate models to overcome this computational challenge. The D-LSPF runs orders of magnitude faster than a high-fidelity particle filter and 3-5 times faster than alternative methods while being up to an order of magnitude more accurate.
• Score: 0.669618059970013
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In Data Assimilation, observations are fused with simulations to obtain an accurate estimate of the state and parameters for a given physical system. Combining data with a model, however, while accurately estimating uncertainty, is computationally expensive and infeasible to run in real-time for complex systems. Here, we present a novel particle filter methodology, the Deep Latent Space Particle filter or D-LSPF, that uses neural network-based surrogate models to overcome this computational challenge. The D-LSPF enables filtering in the low-dimensional latent space obtained using Wasserstein AEs with modified vision transformer layers for dimensionality reduction and transformers for parameterized latent space time stepping. As we demonstrate on three test cases, including leak localization in multi-phase pipe flow and seabed identification for fully nonlinear water waves, the D-LSPF runs orders of magnitude faster than a high-fidelity particle filter and 3-5 times faster than alternative methods while being up to an order of magnitude more accurate. The D-LSPF thus enables real-time data assimilation with uncertainty quantification for physical systems.

Related papers

• TorchDA: A Python package for performing data assimilation with deep learning forward and transformation functions [7.11946037942663]
  This study presents a novel Python package combining data assimilation with deep neural networks to serve as models for state transition and observation functions. Comprehensive experiments conducted on the Lorenz 63 and a two-dimensional shallow water system demonstrate significantly enhanced performance over standalone model predictions without assimilation. Overall, this innovative software package enables flexible integration of deep learning representations within data assimilation, conferring a versatile tool to tackle complex high dimensional dynamical systems across scientific domains.
  arXiv  Detail & Related papers  (2024-08-30T20:30:34Z)
• Aero-Nef: Neural Fields for Rapid Aircraft Aerodynamics Simulations [1.1932047172700866]
  This paper presents a methodology to learn surrogate models of steady state fluid dynamics simulations on meshed domains. The proposed models can be applied directly to unstructured domains for different flow conditions. Remarkably, the method can perform inference five order of magnitude faster than the high fidelity solver on the RANS transonic airfoil dataset.
  arXiv  Detail & Related papers  (2024-07-29T11:48:44Z)
• Closed-form Filtering for Non-linear Systems [83.91296397912218]
  We propose a new class of filters based on Gaussian PSD Models, which offer several advantages in terms of density approximation and computational efficiency. We show that filtering can be efficiently performed in closed form when transitions and observations are Gaussian PSD Models. Our proposed estimator enjoys strong theoretical guarantees, with estimation error that depends on the quality of the approximation and is adaptive to the regularity of the transition probabilities.
  arXiv  Detail & Related papers  (2024-02-15T08:51:49Z)
• An Ensemble Score Filter for Tracking High-Dimensional Nonlinear Dynamical Systems [10.997994515823798]
  We propose an ensemble score filter (EnSF) for solving high-dimensional nonlinear filtering problems. Unlike existing diffusion models that train neural networks to approximate the score function, we develop a training-free score estimation. EnSF provides surprising performance, compared with the state-of-the-art Local Ensemble Transform Kalman Filter method.
  arXiv  Detail & Related papers  (2023-09-02T16:48:02Z)
• Learning IMM Filter Parameters from Measurements using Gradient Descent [45.335821132209766]
  In intrinsic parameters of targets under track can be completely unobservable until the system is deployed. With state-of-the-art sensor systems growing more and more complex, the number of parameters naturally increases. In this paper, the parameters of an interacting multiple model (IMM) filter are optimized solely using measurements.
  arXiv  Detail & Related papers  (2023-07-13T08:35:40Z)
• Filter Pruning for Efficient CNNs via Knowledge-driven Differential Filter Sampler [103.97487121678276]
  Filter pruning simultaneously accelerates the computation and reduces the memory overhead of CNNs. We propose a novel Knowledge-driven Differential Filter Sampler(KDFS) with Masked Filter Modeling(MFM) framework for filter pruning.
  arXiv  Detail & Related papers  (2023-07-01T02:28:41Z)
• Dynamic Temporal Filtering in Video Models [128.02725199486719]
  We present a new recipe of temporal feature learning, namely Dynamic Temporal Filter (DTF) DTF learns a specialized frequency filter for every spatial location to model its long-range temporal dynamics. It is feasible to plug DTF block into ConvNets and Transformer, yielding DTF-Net and DTF-Transformer.
  arXiv  Detail & Related papers  (2022-11-15T15:59:28Z)
• Computational Doob's h-transforms for Online Filtering of Discretely Observed Diffusions [65.74069050283998]
  We propose a computational framework to approximate Doob's $h$-transforms. The proposed approach can be orders of magnitude more efficient than state-of-the-art particle filters.
  arXiv  Detail & Related papers  (2022-06-07T15:03:05Z)
• An efficient estimation of time-varying parameters of dynamic models by combining offline batch optimization and online data assimilation [0.0]
  I present an efficient and practical method to estimate the time-varying parameters of relatively low dimensional models. I propose combining offline batch optimization and online data assimilation.
  arXiv  Detail & Related papers  (2021-10-24T20:12:12Z)
• Adaptive Machine Learning for Time-Varying Systems: Low Dimensional Latent Space Tuning [91.3755431537592]
  We present a recently developed method of adaptive machine learning for time-varying systems. Our approach is to map very high (N>100k) dimensional inputs into the low dimensional (N2) latent space at the output of the encoder section of an encoder-decoder CNN. This method allows us to learn correlations within and to track their evolution in real time based on feedback without interrupts.
  arXiv  Detail & Related papers  (2021-07-13T16:05:28Z)
• Adaptive Latent Space Tuning for Non-Stationary Distributions [62.997667081978825]
  We present a method for adaptive tuning of the low-dimensional latent space of deep encoder-decoder style CNNs. We demonstrate our approach for predicting the properties of a time-varying charged particle beam in a particle accelerator.
  arXiv  Detail & Related papers  (2021-05-08T03:50:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.