Transformers Can Solve Non-Linear and Non-Markovian Filtering Problems in Continuous Time For Conditionally Gaussian Signals

• URL: http://arxiv.org/abs/2310.19603v4
• Date: Mon, 14 Jul 2025 13:17:47 GMT
• Title: Transformers Can Solve Non-Linear and Non-Markovian Filtering Problems in Continuous Time For Conditionally Gaussian Signals
• Authors: Blanka Horvath, Anastasis Kratsios, Yannick Limmer, Xuwei Yang,
• Abstract summary: We show that a class of continuous-time transformer models, called textitfilterformers, can approximately implement the conditional law of a broad class of non-Markovian and conditionally Gaussian signal processes.<n>Our approximation guarantees hold uniformly over sufficiently regular compact subsets of continuous-time paths, where the worst-case 2-Wasserstein distance between the true optimal filter and our deep learning model quantifies the approximation error.
• Score: 8.858600521363293
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The use of attention-based deep learning models in stochastic filtering, e.g. transformers and deep Kalman filters, has recently come into focus; however, the potential for these models to solve stochastic filtering problems remains largely unknown. The paper provides an affirmative answer to this open problem in the theoretical foundations of machine learning by showing that a class of continuous-time transformer models, called \textit{filterformers}, can approximately implement the conditional law of a broad class of non-Markovian and conditionally Gaussian signal processes given noisy continuous-time (possibly non-Gaussian) measurements. Our approximation guarantees hold uniformly over sufficiently regular compact subsets of continuous-time paths, where the worst-case 2-Wasserstein distance between the true optimal filter and our deep learning model quantifies the approximation error. Our construction relies on two new customizations of the standard attention mechanism: The first can losslessly adapt to the characteristics of a broad range of paths since we show that the attention mechanism implements bi-Lipschitz embeddings of sufficiently regular sets of paths into low-dimensional Euclidean spaces; thus, it incurs no ``dimension reduction error''. The latter attention mechanism is tailored to the geometry of Gaussian measures in the $2$-Wasserstein space. Our analysis relies on new stability estimates of robust optimal filters in the conditionally Gaussian setting.

Related papers

• A competitive baseline for deep learning enhanced data assimilation using conditional Gaussian ensemble Kalman filtering [0.0]
  We study two non-linear extensions of the vanilla EnKF, dubbed the conditional-Gaussian EnKF (CG-EnKF) and the normal score EnKF (NS-EnKF) We compare these models against a state-of-the-art deep learning based particle filter called the score filter (SF) Our analysis also demonstrates that the CG-EnKF and NS-EnKF can handle highly non-Gaussian additive noise perturbations, with the latter typically outperforming the former.
  arXiv  Detail & Related papers  (2024-09-22T02:54:33Z)
• Learning Optimal Filters Using Variational Inference [0.3749861135832072]
  We present a framework for learning a parameterized analysis map for filtering systems. In principle this can lead to a better approximation of the filtering distribution, and hence smaller bias. We show that this methodology can be used to learn the gain matrix, in an affine analysis map, for filtering linear and nonlinear dynamical systems.
  arXiv  Detail & Related papers  (2024-06-26T04:51:14Z)
• Bayesian Circular Regression with von Mises Quasi-Processes [57.88921637944379]
  In this work we explore a family of expressive and interpretable distributions over circle-valued random functions.<n>For posterior inference, we introduce a new Stratonovich-like augmentation that lends itself to fast Gibbs sampling.<n>We present experiments applying this model to the prediction of wind directions and the percentage of the running gait cycle as a function of joint angles.
  arXiv  Detail & Related papers  (2024-06-19T01:57:21Z)
• Outlier-robust Kalman Filtering through Generalised Bayes [45.51425214486509]
  We derive a novel, provably robust, and closed-form Bayesian update rule for online filtering in state-space models. Our method matches or outperforms other robust filtering methods at a much lower computational cost.
  arXiv  Detail & Related papers  (2024-05-09T09:40:56Z)
• Closed-form Filtering for Non-linear Systems [83.91296397912218]
  We propose a new class of filters based on Gaussian PSD Models, which offer several advantages in terms of density approximation and computational efficiency. We show that filtering can be efficiently performed in closed form when transitions and observations are Gaussian PSD Models. Our proposed estimator enjoys strong theoretical guarantees, with estimation error that depends on the quality of the approximation and is adaptive to the regularity of the transition probabilities.
  arXiv  Detail & Related papers  (2024-02-15T08:51:49Z)
• Pathspace Kalman Filters with Dynamic Process Uncertainty for Analyzing Time-course Data [4.350285695981938]
  We develop a Pathspace Kalman Filter (PKF) which allows us to track the uncertainties associated with the underlying data and prior knowledge. An application of this algorithm is to automatically detect temporal windows where the internal mechanistic model deviates from the data in a time-dependent manner. We numerically demonstrate that the PKF outperforms conventional KF methods on a synthetic dataset lowering the mean-squared-error by several orders of magnitude.
  arXiv  Detail & Related papers  (2024-02-07T00:54:35Z)
• Fast sampling from constrained spaces using the Metropolis-adjusted Mirror Langevin algorithm [12.405427902037971]
  We propose a new method for approximate sampling from distributions whose support is a compact and convex set. This algorithm adds an accept-reject filter to the Markov chain induced by a single step of the Mirror Langevin. We obtain an exponentially better dependence on the error tolerance for approximate constrained sampling.
  arXiv  Detail & Related papers  (2023-12-14T11:11:58Z)
• Nonlinear Filtering with Brenier Optimal Transport Maps [4.745059103971596]
  This paper is concerned with the problem of nonlinear filtering, i.e., computing the conditional distribution of the state of a dynamical system. Conventional sequential importance resampling (SIR) particle filters suffer from fundamental limitations, in scenarios involving degenerate likelihoods or high-dimensional states. In this paper, we explore an alternative method, which is based on estimating the Brenier optimal transport (OT) map from the current prior distribution of the state to the posterior distribution at the next time step.
  arXiv  Detail & Related papers  (2023-10-21T01:34:30Z)
• Outlier-Insensitive Kalman Filtering Using NUV Priors [24.413595920205907]
  In practice, observations are corrupted by outliers, severely impairing the Kalman filter (KF)s performance. In this work, an outlier-insensitive KF is proposed, where is achieved by modeling each potential outlier as a normally distributed random variable with unknown variance (NUV) The NUVs variances are estimated online, using both expectation-maximization (EM) and alternating robustness (AM)
  arXiv  Detail & Related papers  (2022-10-12T11:00:13Z)
• Beyond the Edge of Stability via Two-step Gradient Updates [49.03389279816152]
  Gradient Descent (GD) is a powerful workhorse of modern machine learning. GD's ability to find local minimisers is only guaranteed for losses with Lipschitz gradients. This work focuses on simple, yet representative, learning problems via analysis of two-step gradient updates.
  arXiv  Detail & Related papers  (2022-06-08T21:32:50Z)
• Computational Doob's h-transforms for Online Filtering of Discretely Observed Diffusions [65.74069050283998]
  We propose a computational framework to approximate Doob's $h$-transforms. The proposed approach can be orders of magnitude more efficient than state-of-the-art particle filters.
  arXiv  Detail & Related papers  (2022-06-07T15:03:05Z)
• Improved Convergence Rate of Stochastic Gradient Langevin Dynamics with Variance Reduction and its Application to Optimization [50.83356836818667]
  gradient Langevin Dynamics is one of the most fundamental algorithms to solve non-eps optimization problems. In this paper, we show two variants of this kind, namely the Variance Reduced Langevin Dynamics and the Recursive Gradient Langevin Dynamics.
  arXiv  Detail & Related papers  (2022-03-30T11:39:00Z)
• Deep Learning for the Benes Filter [91.3755431537592]
  We present a new numerical method based on the mesh-free neural network representation of the density of the solution of the Benes model. We discuss the role of nonlinearity in the filtering model equations for the choice of the domain of the neural network.
  arXiv  Detail & Related papers  (2022-03-09T14:08:38Z)
• A Logarithmic Bayesian Approach to Quantum Error Detection [0.0]
  We propose a pair of digital filters using logarithmic probabilities to achieve near-optimal performance on a three-qubit bit-flip code. These filters are approximations of an optimal filter that we derive explicitly for finite time steps. We demonstrate that the single-term and two-term filters are able to significantly outperform both a double threshold scheme and a linearized version of the Wonham filter.
  arXiv  Detail & Related papers  (2021-10-20T19:08:20Z)
• KalmanNet: Neural Network Aided Kalman Filtering for Partially Known Dynamics [84.18625250574853]
  We present KalmanNet, a real-time state estimator that learns from data to carry out Kalman filtering under non-linear dynamics. We numerically demonstrate that KalmanNet overcomes nonlinearities and model mismatch, outperforming classic filtering methods.
  arXiv  Detail & Related papers  (2021-07-21T12:26:46Z)
• Neural Kalman Filtering [62.997667081978825]
  We show that a gradient-descent approximation to the Kalman filter requires only local computations with variance weighted prediction errors. We also show that it is possible under the same scheme to adaptively learn the dynamics model with a learning rule that corresponds directly to Hebbian plasticity.
  arXiv  Detail & Related papers  (2021-02-19T16:43:15Z)
• Robust Reinforcement Learning with Wasserstein Constraint [49.86490922809473]
  We show the existence of optimal robust policies, provide a sensitivity analysis for the perturbations, and then design a novel robust learning algorithm. The effectiveness of the proposed algorithm is verified in the Cart-Pole environment.
  arXiv  Detail & Related papers  (2020-06-01T13:48:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.