Latent-EnSF: A Latent Ensemble Score Filter for High-Dimensional Data Assimilation with Sparse Observation Data

• URL: http://arxiv.org/abs/2409.00127v3
• Date: Wed, 11 Sep 2024 17:18:58 GMT
• Title: Latent-EnSF: A Latent Ensemble Score Filter for High-Dimensional Data Assimilation with Sparse Observation Data
• Authors: Phillip Si, Peng Chen,
• Abstract summary: We propose a novel data assimilation method, Latent-EnSF, to address the challenges of high dimensionlity in states and high sparsity in observations for nonlinear Bayesian filtering. We demonstrate the higher accuracy, faster convergence, and higher efficiency of Latent-EnSF for two challenging applications with complex models in shallow water wave propagation and medium-range weather forecasting.
• Score: 4.027368782323261
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Accurate modeling and prediction of complex physical systems often rely on data assimilation techniques to correct errors inherent in model simulations. Traditional methods like the Ensemble Kalman Filter (EnKF) and its variants as well as the recently developed Ensemble Score Filters (EnSF) face significant challenges when dealing with high-dimensional and nonlinear Bayesian filtering problems with sparse observations, which are ubiquitous in real-world applications. In this paper, we propose a novel data assimilation method, Latent-EnSF, which leverages EnSF with efficient and consistent latent representations of the full states and sparse observations to address the joint challenges of high dimensionlity in states and high sparsity in observations for nonlinear Bayesian filtering. We introduce a coupled Variational Autoencoder (VAE) with two encoders to encode the full states and sparse observations in a consistent way guaranteed by a latent distribution matching and regularization as well as a consistent state reconstruction. With comparison to several methods, we demonstrate the higher accuracy, faster convergence, and higher efficiency of Latent-EnSF for two challenging applications with complex models in shallow water wave propagation and medium-range weather forecasting, for highly sparse observations in both space and time.

Related papers

• Open-set Anomaly Segmentation in Complex Scenarios [88.11076112792992]
  This paper introduces ComsAmy, a benchmark for open-set anomaly segmentation in complex scenarios. ComsAmy encompasses a wide spectrum of adverse weather conditions, dynamic driving environments, and diverse anomaly types. We propose a novel energy-entropy learning (EEL) strategy that integrates the complementary information from energy and entropy.
  arXiv  Detail & Related papers  (2025-04-28T12:00:10Z)
• Consistency-aware Self-Training for Iterative-based Stereo Matching [13.079759982779013]
  We propose a consistency-aware self-training framework for iterative-based stereo matching. We first observe that regions with larger errors tend to exhibit more pronounced oscillation characteristics during model prediction. We introduce a consistency-aware soft-weighted loss to adjust the weight of pseudo-labels accordingly, relieving the error accumulation and performance degradation problem.
  arXiv  Detail & Related papers  (2025-03-31T05:58:25Z)
• Language-Informed Hyperspectral Image Synthesis for Imbalanced-Small Sample Classification via Semi-Supervised Conditional Diffusion Model [1.9746060146273674]
  This paper proposes Txt2HSI-LDM(VAE), a novel language-informed hyperspectral image synthesis method. To address the high-dimensionality of hyperspectral data, a universal variational autoencoder (VAE) is designed to map the data into a low-dimensional latent space. VAE decodes HSI from latent space generated by the diffusion model with the language conditions as input.
  arXiv  Detail & Related papers  (2025-02-27T02:35:49Z)
• Flow-based Bayesian filtering for high-dimensional nonlinear stochastic dynamical systems [4.382988524355736]
  We propose a flow-based Bayesian filter (FBF) that integrates normalizing flows to construct a novel latent linear state-space model with Gaussian filtering distributions. This framework facilitates efficient density estimation and sampling using invertible transformations provided by normalizing flows. Numerical experiments demonstrate the superior accuracy and efficiency of FBF.
  arXiv  Detail & Related papers  (2025-02-22T14:04:23Z)
• TrackDiffuser: Nearly Model-Free Bayesian Filtering with Diffusion Model [23.40376181606577]
  We present TrackDiffuser, a generative framework addressing both challenges by reformulating Bayesian filtering as a conditional diffusion model. Our approach implicitly learns system dynamics from data to mitigate the effects of inaccurate SSM. TrackDiffuser exhibits remarkable robustness to SSM inaccuracies, offering a practical solution for real-world state estimation problems.
  arXiv  Detail & Related papers  (2025-02-08T16:21:18Z)
• LD-EnSF: Synergizing Latent Dynamics with Ensemble Score Filters for Fast Data Assimilation with Sparse Observations [3.583387803440918]
  We introduce Latent Dynamics EnSF (LD-EnSF), a novel methodology that completely avoids the full dynamics evolution. We also propose a new method for encoding sparse observations into the latent space using Long Short-Term Memory (LSTM) networks.
  arXiv  Detail & Related papers  (2024-11-28T18:27:11Z)
• Spatially-Aware Diffusion Models with Cross-Attention for Global Field Reconstruction with Sparse Observations [1.371691382573869]
  We develop and enhance score-based diffusion models in field reconstruction tasks. We introduce a condition encoding approach to construct a tractable mapping mapping between observed and unobserved regions. We demonstrate the ability of the model to capture possible reconstructions and improve the accuracy of fused results.
  arXiv  Detail & Related papers  (2024-08-30T19:46:23Z)
• SFANet: Spatial-Frequency Attention Network for Weather Forecasting [54.470205739015434]
  Weather forecasting plays a critical role in various sectors, driving decision-making and risk management. Traditional methods often struggle to capture the complex dynamics of meteorological systems. We propose a novel framework designed to address these challenges and enhance the accuracy of weather prediction.
  arXiv  Detail & Related papers  (2024-05-29T08:00:15Z)
• Closed-form Filtering for Non-linear Systems [83.91296397912218]
  We propose a new class of filters based on Gaussian PSD Models, which offer several advantages in terms of density approximation and computational efficiency. We show that filtering can be efficiently performed in closed form when transitions and observations are Gaussian PSD Models. Our proposed estimator enjoys strong theoretical guarantees, with estimation error that depends on the quality of the approximation and is adaptive to the regularity of the transition probabilities.
  arXiv  Detail & Related papers  (2024-02-15T08:51:49Z)
• Take History as a Mirror in Heterogeneous Federated Learning [9.187993085263209]
  Federated Learning (FL) allows several clients to cooperatively train machine learning models without disclosing the raw data. In this work, we propose a novel asynchronous FL framework called Federated Historical Learning (FedHist) FedHist effectively addresses the challenges posed by both Non-IID data and gradient staleness.
  arXiv  Detail & Related papers  (2023-12-16T11:40:49Z)
• Complexity Matters: Rethinking the Latent Space for Generative Modeling [65.64763873078114]
  In generative modeling, numerous successful approaches leverage a low-dimensional latent space, e.g., Stable Diffusion. In this study, we aim to shed light on this under-explored topic by rethinking the latent space from the perspective of model complexity.
  arXiv  Detail & Related papers  (2023-07-17T07:12:29Z)
• Deep learning-enhanced ensemble-based data assimilation for high-dimensional nonlinear dynamical systems [0.0]
  Ensemble Kalman filter (EnKF) is a DA algorithm widely used in applications involving high-dimensional nonlinear dynamical systems. In this work, we propose hybrid ensemble Kalman filter (H-EnKF), which is applied to a two-layer quasi-geostrophic flow system as a test case.
  arXiv  Detail & Related papers  (2022-06-09T23:34:49Z)
• Distributed Dynamic Safe Screening Algorithms for Sparse Regularization [73.85961005970222]
  We propose a new distributed dynamic safe screening (DDSS) method for sparsity regularized models and apply it on shared-memory and distributed-memory architecture respectively. We prove that the proposed method achieves the linear convergence rate with lower overall complexity and can eliminate almost all the inactive features in a finite number of iterations almost surely.
  arXiv  Detail & Related papers  (2022-04-23T02:45:55Z)
• CAFE: Learning to Condense Dataset by Aligning Features [72.99394941348757]
  We propose a novel scheme to Condense dataset by Aligning FEatures (CAFE) At the heart of our approach is an effective strategy to align features from the real and synthetic data across various scales. We validate the proposed CAFE across various datasets, and demonstrate that it generally outperforms the state of the art.
  arXiv  Detail & Related papers  (2022-03-03T05:58:49Z)
• Deep Variational Models for Collaborative Filtering-based Recommender Systems [63.995130144110156]
  Deep learning provides accurate collaborative filtering models to improve recommender system results. Our proposed models apply the variational concept to injectity in the latent space of the deep architecture. Results show the superiority of the proposed approach in scenarios where the variational enrichment exceeds the injected noise effect.
  arXiv  Detail & Related papers  (2021-07-27T08:59:39Z)
• Machine learning-based conditional mean filter: a generalization of the ensemble Kalman filter for nonlinear data assimilation [42.60602838972598]
  We propose a machine learning-based ensemble conditional mean filter (ML-EnCMF) for tracking possibly high-dimensional non-Gaussian state models with nonlinear dynamics based on sparse observations. The proposed filtering method is developed based on the conditional expectation and numerically implemented using machine learning (ML) techniques combined with the ensemble method.
  arXiv  Detail & Related papers  (2021-06-15T06:40:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.