Ensemble Controlled-Flow Filtering for Implicit Data Assimilation

• URL: http://arxiv.org/abs/2607.12975v1
• Date: Tue, 14 Jul 2026 17:16:37 GMT
• Title: Ensemble Controlled-Flow Filtering for Implicit Data Assimilation
• Abstract summary: We introduce implicit data assimilation, in which the analysis law is defined as an energy tilt of the forecast distribution.<n>We then propose the Ensemble Controlled-flow Filter (EnCF), which realizes this update through a controlled flow.<n>For simulator-defined observations, EnCF-LF learns a surrogate conditional energy from samples and applies the same controlled-flow solver.
• Score: 11.531404987674378
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Data assimilation estimates the state of a dynamical system from model forecasts and incoming observations. Many observation mechanisms, however, are many-to-one, implicit, non-smooth, or accessible only through simulation, and need not provide the residual structures or likelihood guidance required by existing ensemble filters. We introduce implicit data assimilation, in which the analysis law is defined as an energy tilt of the forecast distribution. We then propose the Ensemble Controlled-flow Filter (EnCF), which realizes this update through a stochastic controlled flow and learns the observation-dependent control by adjoint matching from terminal energy gradients. For simulator-defined observations, EnCF-LF learns a surrogate conditional energy from samples and applies the same controlled-flow solver. We prove ideal exactness, derive a one-step error decomposition, and establish non-accumulation of local errors under filter stability. Numerical results show that Kalman-type filters remain preferable for smooth additive-Gaussian observations, while the proposed methods are better suited to non-Gaussian, many-to-one, multimodal, and implicit observation models.

Related papers

• Generative Bayesian Filtering for State Estimation [8.548885512840073]
  We propose a filtering framework that replaces restrictive observation models with pretrained conditional generative models parametrized by conditional variational autoencoders (CVAE)<n>Experiments on synthetic datasets and real-world applications involving manufacturing system monitoring and arrhythmia diagnosis demonstrate that GBF improves state estimation accuracy and robustness relative to baseline approaches.
  arXiv  Detail & Related papers  (2026-07-09T00:05:23Z)
• Learning Probabilistic Filters with Strictly Proper Scoring Rules [8.61813579171868]
  We introduce the proper scoring ensemble filter (PSEF), an ensemble data assimilation method based on training an analysis map to approximate the filtering distribution.<n>We prove that, under a realizability assumption, the population objective is minimized by the true Bayesian filtering distribution.<n> Numerical experiments show that the learned filter accurately approximates challenging filtering distributions, including nonlinear, non-Gaussian, and multi-modal posteriors.
  arXiv  Detail & Related papers  (2026-06-25T01:04:21Z)
• DAISI: Data Assimilation with Inverse Sampling using Stochastic Interpolants [12.587156528707796]
  We introduce DAISI, a scalable filtering algorithm built on flow-based generative models.<n>We show that DAISI achieves accurate filtering results in regimes with sparse, noisy, and nonlinear observations.
  arXiv  Detail & Related papers  (2025-11-29T00:02:45Z)
• Filtering with Randomised Observations: Sequential Learning of Relevant Subspace Properties and Accuracy Analysis [0.0]
  We examine the signal-tracking performance of a continuous ensemble Kalman filtering under fixed, randomised, and adaptively varying partial observations.<n>We propose a sequential learning scheme that adaptively determines the dimension of a state subspace sufficient to ensure bounded filtering error.
  arXiv  Detail & Related papers  (2025-09-05T07:30:01Z)
• Feynman-Kac Correctors in Diffusion: Annealing, Guidance, and Product of Experts [64.34482582690927]
  We provide an efficient and principled method for sampling from a sequence of annealed, geometric-averaged, or product distributions derived from pretrained score-based models.<n>We propose Sequential Monte Carlo (SMC) resampling algorithms that leverage inference-time scaling to improve sampling quality.
  arXiv  Detail & Related papers  (2025-03-04T17:46:51Z)
• Closed-form Filtering for Non-linear Systems [83.91296397912218]
  We propose a new class of filters based on Gaussian PSD Models, which offer several advantages in terms of density approximation and computational efficiency. We show that filtering can be efficiently performed in closed form when transitions and observations are Gaussian PSD Models. Our proposed estimator enjoys strong theoretical guarantees, with estimation error that depends on the quality of the approximation and is adaptive to the regularity of the transition probabilities.
  arXiv  Detail & Related papers  (2024-02-15T08:51:49Z)
• An adaptive ensemble filter for heavy-tailed distributions: tuning-free inflation and localization [0.3749861135832072]
  Heavy tails is a common feature of filtering distributions that results from the nonlinear dynamical and observation processes. We propose an algorithm to estimate the prior-to-posterior update from samples of joint forecast distribution of the states and observations. We demonstrate the benefits of this new ensemble filter on challenging filtering problems.
  arXiv  Detail & Related papers  (2023-10-12T21:56:14Z)
• Score-based Continuous-time Discrete Diffusion Models [102.65769839899315]
  We extend diffusion models to discrete variables by introducing a Markov jump process where the reverse process denoises via a continuous-time Markov chain. We show that an unbiased estimator can be obtained via simple matching the conditional marginal distributions. We demonstrate the effectiveness of the proposed method on a set of synthetic and real-world music and image benchmarks.
  arXiv  Detail & Related papers  (2022-11-30T05:33:29Z)
• Convolutional Filtering on Sampled Manifolds [122.06927400759021]
  We show that convolutional filtering on a sampled manifold converges to continuous manifold filtering. Our findings are further demonstrated empirically on a problem of navigation control.
  arXiv  Detail & Related papers  (2022-11-20T19:09:50Z)
• Computational Doob's h-transforms for Online Filtering of Discretely Observed Diffusions [65.74069050283998]
  We propose a computational framework to approximate Doob's $h$-transforms. The proposed approach can be orders of magnitude more efficient than state-of-the-art particle filters.
  arXiv  Detail & Related papers  (2022-06-07T15:03:05Z)
• Machine learning-based conditional mean filter: a generalization of the ensemble Kalman filter for nonlinear data assimilation [42.60602838972598]
  We propose a machine learning-based ensemble conditional mean filter (ML-EnCMF) for tracking possibly high-dimensional non-Gaussian state models with nonlinear dynamics based on sparse observations. The proposed filtering method is developed based on the conditional expectation and numerically implemented using machine learning (ML) techniques combined with the ensemble method.
  arXiv  Detail & Related papers  (2021-06-15T06:40:32Z)
• Semi-Supervised Learning with Normalizing Flows [54.376602201489995]
  FlowGMM is an end-to-end approach to generative semi supervised learning with normalizing flows. We show promising results on a wide range of applications, including AG-News and Yahoo Answers text data.
  arXiv  Detail & Related papers  (2019-12-30T17:36:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.