Pathspace Kalman Filters with Dynamic Process Uncertainty for Analyzing Time-course Data

• URL: http://arxiv.org/abs/2402.04498v2
• Date: Mon, 1 Apr 2024 21:58:10 GMT
• Title: Pathspace Kalman Filters with Dynamic Process Uncertainty for Analyzing Time-course Data
• Authors: Chaitra Agrahar, William Poole, Simone Bianco, Hana El-Samad,
• Abstract summary: We develop a Pathspace Kalman Filter (PKF) which allows us to track the uncertainties associated with the underlying data and prior knowledge. An application of this algorithm is to automatically detect temporal windows where the internal mechanistic model deviates from the data in a time-dependent manner. We numerically demonstrate that the PKF outperforms conventional KF methods on a synthetic dataset lowering the mean-squared-error by several orders of magnitude.
• Score: 4.350285695981938
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Kalman Filter (KF) is an optimal linear state prediction algorithm, with applications in fields as diverse as engineering, economics, robotics, and space exploration. Here, we develop an extension of the KF, called a Pathspace Kalman Filter (PKF) which allows us to a) dynamically track the uncertainties associated with the underlying data and prior knowledge, and b) take as input an entire trajectory and an underlying mechanistic model, and using a Bayesian methodology quantify the different sources of uncertainty. An application of this algorithm is to automatically detect temporal windows where the internal mechanistic model deviates from the data in a time-dependent manner. First, we provide theorems characterizing the convergence of the PKF algorithm. Then, we numerically demonstrate that the PKF outperforms conventional KF methods on a synthetic dataset lowering the mean-squared-error by several orders of magnitude. Finally, we apply this method to biological time-course dataset involving over 1.8 million gene expression measurements.

Related papers

• AI-Aided Kalman Filters [65.35350122917914]
  The Kalman filter (KF) and its variants are among the most celebrated algorithms in signal processing. Recent developments illustrate the possibility of fusing deep neural networks (DNNs) with classic Kalman-type filtering. This article provides a tutorial-style overview of design approaches for incorporating AI in aiding KF-type algorithms.
  arXiv  Detail & Related papers  (2024-10-16T06:47:53Z)
• Deep Kalman Filters Can Filter [9.131190818372474]
  Deep Kalman filters (DKFs) are a class of neural network models that generate Gaussian probability measures from sequential data. DKFs are inspired by the Kalman filter, but they lack concrete theoretical ties to the filtering problem. We show that continuous-time DKFs can implement the conditional law of a broad class of non-Markovian and conditionally Gaussian signal processes.
  arXiv  Detail & Related papers  (2023-10-30T14:58:12Z)
• Large-Scale OD Matrix Estimation with A Deep Learning Method [70.78575952309023]
  The proposed method integrates deep learning and numerical optimization algorithms to infer matrix structure and guide numerical optimization. We conducted tests to demonstrate the good generalization performance of our method on a large-scale synthetic dataset.
  arXiv  Detail & Related papers  (2023-10-09T14:30:06Z)
• Low-rank extended Kalman filtering for online learning of neural networks from streaming data [71.97861600347959]
  We propose an efficient online approximate Bayesian inference algorithm for estimating the parameters of a nonlinear function from a potentially non-stationary data stream. The method is based on the extended Kalman filter (EKF), but uses a novel low-rank plus diagonal decomposition of the posterior matrix. In contrast to methods based on variational inference, our method is fully deterministic, and does not require step-size tuning.
  arXiv  Detail & Related papers  (2023-05-31T03:48:49Z)
• Gaussian process regression and conditional Karhunen-Lo\'{e}ve models for data assimilation in inverse problems [68.8204255655161]
  We present a model inversion algorithm, CKLEMAP, for data assimilation and parameter estimation in partial differential equation models. The CKLEMAP method provides better scalability compared to the standard MAP method.
  arXiv  Detail & Related papers  (2023-01-26T18:14:12Z)
• Manifold Interpolating Optimal-Transport Flows for Trajectory Inference [64.94020639760026]
  We present a method called Manifold Interpolating Optimal-Transport Flow (MIOFlow) MIOFlow learns, continuous population dynamics from static snapshot samples taken at sporadic timepoints. We evaluate our method on simulated data with bifurcations and merges, as well as scRNA-seq data from embryoid body differentiation, and acute myeloid leukemia treatment.
  arXiv  Detail & Related papers  (2022-06-29T22:19:03Z)
• Deep learning-enhanced ensemble-based data assimilation for high-dimensional nonlinear dynamical systems [0.0]
  Ensemble Kalman filter (EnKF) is a DA algorithm widely used in applications involving high-dimensional nonlinear dynamical systems. In this work, we propose hybrid ensemble Kalman filter (H-EnKF), which is applied to a two-layer quasi-geostrophic flow system as a test case.
  arXiv  Detail & Related papers  (2022-06-09T23:34:49Z)
• COSMIC: fast closed-form identification from large-scale data for LTV systems [4.10464681051471]
  We introduce a closed-form method for identification of discrete-time linear timevariant systems from data. We develop an algorithm with guarantees of optimality and with a complexity that increases linearly with the number of instants considered per trajectory. Our algorithm was applied to both a Low Fidelity and Functional Engineering Simulators for the Comet Interceptor mission.
  arXiv  Detail & Related papers  (2021-12-08T16:07:59Z)
• Higher Order Kernel Mean Embeddings to Capture Filtrations of Stochastic Processes [11.277354787690646]
  We introduce a family of higher order kernel mean embeddings that generalizes the notion of KME. We derive empirical estimators for the associated higher order maximum mean discrepancies (MMDs) and prove consistency. We construct a family of universal kernels on processes that allows to solve real-world calibration and optimal stopping problems.
  arXiv  Detail & Related papers  (2021-09-08T12:27:25Z)
• KalmanNet: Neural Network Aided Kalman Filtering for Partially Known Dynamics [84.18625250574853]
  We present KalmanNet, a real-time state estimator that learns from data to carry out Kalman filtering under non-linear dynamics. We numerically demonstrate that KalmanNet overcomes nonlinearities and model mismatch, outperforming classic filtering methods.
  arXiv  Detail & Related papers  (2021-07-21T12:26:46Z)
• Low-Rank Hankel Tensor Completion for Traffic Speed Estimation [7.346671461427793]
  We propose a purely data-driven and model-free solution to the traffic state estimation problem. By imposing a low-rank assumption on this tensor structure, we can approximate characterize both global patterns and the unknown complex local dynamics. We conduct numerical experiments on both synthetic simulation data and real-world high-resolution data, and our results demonstrate the effectiveness and superiority of the proposed model.
  arXiv  Detail & Related papers  (2021-05-21T00:08:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.