Data-Assimilated Model-Based Reinforcement Learning for Partially Observed Chaotic Flows

• URL: http://arxiv.org/abs/2504.16588v1
• Date: Wed, 23 Apr 2025 10:12:53 GMT
• Title: Data-Assimilated Model-Based Reinforcement Learning for Partially Observed Chaotic Flows
• Authors: Defne E. Ozan, Andrea Nóvoa, Luca Magri,
• Abstract summary: We propose a data-assimilated model-based RL (DA-MBRL) framework for systems with partial observability and noisy measurements.<n>An off-policy actor-critic algorithm is employed to learn optimal control strategies from state estimates.<n>The framework is tested on the Kuramoto-Sivainskysh equation, demonstrating its effectiveness in stabilizing atemporally chaotic flow.
• Score: 3.7960472831772765
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The goal of many applications in energy and transport sectors is to control turbulent flows. However, because of chaotic dynamics and high dimensionality, the control of turbulent flows is exceedingly difficult. Model-free reinforcement learning (RL) methods can discover optimal control policies by interacting with the environment, but they require full state information, which is often unavailable in experimental settings. We propose a data-assimilated model-based RL (DA-MBRL) framework for systems with partial observability and noisy measurements. Our framework employs a control-aware Echo State Network for data-driven prediction of the dynamics, and integrates data assimilation with an Ensemble Kalman Filter for real-time state estimation. An off-policy actor-critic algorithm is employed to learn optimal control strategies from state estimates. The framework is tested on the Kuramoto-Sivashinsky equation, demonstrating its effectiveness in stabilizing a spatiotemporally chaotic flow from noisy and partial measurements.

Related papers

• Stochastic Control for Fine-tuning Diffusion Models: Optimality, Regularity, and Convergence [11.400431211239958]
  Diffusion models have emerged as powerful tools for generative modeling. We propose a control framework for fine-tuning diffusion models. We show that PI-FT achieves global convergence at a linear rate.
  arXiv  Detail & Related papers  (2024-12-24T04:55:46Z)
• Latent feedback control of distributed systems in multiple scenarios through deep learning-based reduced order models [3.5161229331588095]
  Continuous monitoring and real-time control of high-dimensional distributed systems are crucial in applications to ensure a desired physical behavior.<n>Traditional feedback control design that relies on full-order models fails to meet these requirements due to the delay in the control computation.<n>We propose a real-time closed-loop control strategy enhanced by nonlinear non-intrusive Deep Learning-based Reduced Order Models (DL-ROMs)
  arXiv  Detail & Related papers  (2024-12-13T08:04:21Z)
• Amortized Control of Continuous State Space Feynman-Kac Model for Irregular Time Series [14.400596021890863]
  Many real-world datasets, such as healthcare, climate, and economics, are often collected as irregular time series.<n>We propose the Amortized Control of continuous State Space Model (ACSSM) for continuous dynamical modeling of time series.
  arXiv  Detail & Related papers  (2024-10-08T01:27:46Z)
• Learning Noise-Robust Stable Koopman Operator for Control with Hankel DMD [1.0742675209112622]
  We propose a noise-robust learning framework for the Koopman operator of nonlinear dynamical systems.<n>We leverage observables generated by the system dynamics, when the system dynamics is known, through a Hankel matrix.<n>We approximate them with a neural network while maintaining structural similarities to discrete Polyflow.
  arXiv  Detail & Related papers  (2024-08-13T03:39:34Z)
• Semi-Supervised Model-Free Bayesian State Estimation from Compressed Measurements [57.04370580292727]
  We consider data-driven Bayesian state estimation from compressed measurements.<n>The dimension of the temporal measurement vector is lower than that of the temporal state vector to be estimated.<n>The underlying dynamical model of the state's evolution is unknown for a'model-free process'
  arXiv  Detail & Related papers  (2024-07-10T05:03:48Z)
• In-Distribution Barrier Functions: Self-Supervised Policy Filters that Avoid Out-of-Distribution States [84.24300005271185]
  We propose a control filter that wraps any reference policy and effectively encourages the system to stay in-distribution with respect to offline-collected safe demonstrations. Our method is effective for two different visuomotor control tasks in simulation environments, including both top-down and egocentric view settings.
  arXiv  Detail & Related papers  (2023-01-27T22:28:19Z)
• Data efficient reinforcement learning and adaptive optimal perimeter control of network traffic dynamics [0.0]
  This work proposes an integral reinforcement learning (IRL) based approach to learning the macroscopic traffic dynamics for adaptive optimal perimeter control. To reduce the sampling complexity and use the available data more efficiently, the experience replay (ER) technique is introduced to the IRL algorithm. The convergence of the IRL-based algorithms and the stability of the controlled traffic dynamics are proven via the Lyapunov theory.
  arXiv  Detail & Related papers  (2022-09-13T04:28:49Z)
• Learning Robust Output Control Barrier Functions from Safe Expert Demonstrations [50.37808220291108]
  This paper addresses learning safe output feedback control laws from partial observations of expert demonstrations. We first propose robust output control barrier functions (ROCBFs) as a means to guarantee safety. We then formulate an optimization problem to learn ROCBFs from expert demonstrations that exhibit safe system behavior.
  arXiv  Detail & Related papers  (2021-11-18T23:21:00Z)
• Dream to Explore: Adaptive Simulations for Autonomous Systems [3.0664963196464448]
  We tackle the problem of learning to control dynamical systems by applying Bayesian nonparametric methods. By employing Gaussian processes to discover latent world dynamics, we mitigate common data efficiency issues observed in reinforcement learning. Our algorithm jointly learns a world model and policy by optimizing a variational lower bound of a log-likelihood.
  arXiv  Detail & Related papers  (2021-10-27T04:27:28Z)
• KalmanNet: Neural Network Aided Kalman Filtering for Partially Known Dynamics [84.18625250574853]
  We present KalmanNet, a real-time state estimator that learns from data to carry out Kalman filtering under non-linear dynamics. We numerically demonstrate that KalmanNet overcomes nonlinearities and model mismatch, outperforming classic filtering methods.
  arXiv  Detail & Related papers  (2021-07-21T12:26:46Z)
• Robust Value Iteration for Continuous Control Tasks [99.00362538261972]
  When transferring a control policy from simulation to a physical system, the policy needs to be robust to variations in the dynamics to perform well. We present Robust Fitted Value Iteration, which uses dynamic programming to compute the optimal value function on the compact state domain. We show that robust value is more robust compared to deep reinforcement learning algorithm and the non-robust version of the algorithm.
  arXiv  Detail & Related papers  (2021-05-25T19:48:35Z)
• Information Theoretic Model Predictive Q-Learning [64.74041985237105]
  We present a novel theoretical connection between information theoretic MPC and entropy regularized RL. We develop a Q-learning algorithm that can leverage biased models.
  arXiv  Detail & Related papers  (2019-12-31T00:29:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.