Auto-differentiable Ensemble Kalman Filters

• URL: http://arxiv.org/abs/2107.07687v2
• Date: Mon, 19 Jul 2021 18:32:48 GMT
• Title: Auto-differentiable Ensemble Kalman Filters
• Authors: Yuming Chen, Daniel Sanz-Alonso, Rebecca Willett
• Abstract summary: This paper introduces a machine learning framework for learning dynamical systems in data assimilation. Our auto-differentiable ensemble Kalman filters (AD-EnKFs) blend ensemble Kalman filters for state recovery with machine learning tools for learning the dynamics.
• Score: 21.325532465498913
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Data assimilation is concerned with sequentially estimating a temporally-evolving state. This task, which arises in a wide range of scientific and engineering applications, is particularly challenging when the state is high-dimensional and the state-space dynamics are unknown. This paper introduces a machine learning framework for learning dynamical systems in data assimilation. Our auto-differentiable ensemble Kalman filters (AD-EnKFs) blend ensemble Kalman filters for state recovery with machine learning tools for learning the dynamics. In doing so, AD-EnKFs leverage the ability of ensemble Kalman filters to scale to high-dimensional states and the power of automatic differentiation to train high-dimensional surrogate models for the dynamics. Numerical results using the Lorenz-96 model show that AD-EnKFs outperform existing methods that use expectation-maximization or particle filters to merge data assimilation and machine learning. In addition, AD-EnKFs are easy to implement and require minimal tuning.

Related papers

• AI-Aided Kalman Filters [65.35350122917914]
  The Kalman filter (KF) and its variants are among the most celebrated algorithms in signal processing. Recent developments illustrate the possibility of fusing deep neural networks (DNNs) with classic Kalman-type filtering. This article provides a tutorial-style overview of design approaches for incorporating AI in aiding KF-type algorithms.
  arXiv  Detail & Related papers  (2024-10-16T06:47:53Z)
• Uncertainty Representations in State-Space Layers for Deep Reinforcement Learning under Partial Observability [59.758009422067]
  We propose a standalone Kalman filter layer that performs closed-form Gaussian inference in linear state-space models. Similar to efficient linear recurrent layers, the Kalman filter layer processes sequential data using a parallel scan. Experiments show that Kalman filter layers excel in problems where uncertainty reasoning is key for decision-making, outperforming other stateful models.
  arXiv  Detail & Related papers  (2024-09-25T11:22:29Z)
• KFD-NeRF: Rethinking Dynamic NeRF with Kalman Filter [49.85369344101118]
  We introduce KFD-NeRF, a novel dynamic neural radiance field integrated with an efficient and high-quality motion reconstruction framework based on Kalman filtering. Our key idea is to model the dynamic radiance field as a dynamic system whose temporally varying states are estimated based on two sources of knowledge: observations and predictions. Our KFD-NeRF demonstrates similar or even superior performance within comparable computational time and state-of-the-art view synthesis performance with thorough training.
  arXiv  Detail & Related papers  (2024-07-18T05:48:24Z)
• Reduced-Order Autodifferentiable Ensemble Kalman Filters [15.203199227862477]
  We learn a latent low-dimensional surrogate model for the dynamics and a decoder that maps from the latent space to the state space. The learned dynamics and decoder are then used within an ensemble Kalman filter to reconstruct and forecast the state.
  arXiv  Detail & Related papers  (2023-01-27T19:32:02Z)
• Deep learning-enhanced ensemble-based data assimilation for high-dimensional nonlinear dynamical systems [0.0]
  Ensemble Kalman filter (EnKF) is a DA algorithm widely used in applications involving high-dimensional nonlinear dynamical systems. In this work, we propose hybrid ensemble Kalman filter (H-EnKF), which is applied to a two-layer quasi-geostrophic flow system as a test case.
  arXiv  Detail & Related papers  (2022-06-09T23:34:49Z)
• Gradient-Based Trajectory Optimization With Learned Dynamics [80.41791191022139]
  We use machine learning techniques to learn a differentiable dynamics model of the system from data. We show that a neural network can model highly nonlinear behaviors accurately for large time horizons. In our hardware experiments, we demonstrate that our learned model can represent complex dynamics for both the Spot and Radio-controlled (RC) car.
  arXiv  Detail & Related papers  (2022-04-09T22:07:34Z)
• KalmanNet: Neural Network Aided Kalman Filtering for Partially Known Dynamics [84.18625250574853]
  We present KalmanNet, a real-time state estimator that learns from data to carry out Kalman filtering under non-linear dynamics. We numerically demonstrate that KalmanNet overcomes nonlinearities and model mismatch, outperforming classic filtering methods.
  arXiv  Detail & Related papers  (2021-07-21T12:26:46Z)
• Using Data Assimilation to Train a Hybrid Forecast System that Combines Machine-Learning and Knowledge-Based Components [52.77024349608834]
  We consider the problem of data-assisted forecasting of chaotic dynamical systems when the available data is noisy partial measurements. We show that by using partial measurements of the state of the dynamical system, we can train a machine learning model to improve predictions made by an imperfect knowledge-based model.
  arXiv  Detail & Related papers  (2021-02-15T19:56:48Z)
• Online learning of both state and dynamics using ensemble Kalman filters [0.0]
  This paper investigates the possibility to learn both the dynamics and the state online, i.e. to update their estimates at any time. We consider the implication of learning dynamics online through (i) a global EnKF, (i) a local EnKF and (iii) an iterative EnKF. We then demonstrate numerically the efficiency and assess the accuracy of these methods using one-dimensional, one-scale and two-scale chaotic Lorenz models.
  arXiv  Detail & Related papers  (2020-06-06T13:19:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.