Related papers: Data-assimilated model-informed reinforcement learning

Data-assimilated model-informed reinforcement learning

URL: http://arxiv.org/abs/2506.01755v2
Date: Tue, 03 Jun 2025 09:22:29 GMT
Title: Data-assimilated model-informed reinforcement learning
Authors: Defne E. Ozan, Andrea Nóvoa, Georgios Rigas, Luca Magri,
Abstract summary: In practice, sensors often provide only partial and noisy measurements (obations) of the system.<n>We propose a framework that enables the control of chaotic systems with partial and noisy observability.<n>We show that DA-MIRL successfully estimates and suppresses the chaotic dynamics of the environment in real time from partial observations and approximate models.
Score: 3.4748713192043876
License: http://creativecommons.org/licenses/by/4.0/
Abstract: The control of spatio-temporally chaos is challenging because of high dimensionality and unpredictability. Model-free reinforcement learning (RL) discovers optimal control policies by interacting with the system, typically requiring observations of the full physical state. In practice, sensors often provide only partial and noisy measurements (observations) of the system. The objective of this paper is to develop a framework that enables the control of chaotic systems with partial and noisy observability. The proposed method, data-assimilated model-informed reinforcement learning (DA-MIRL), integrates (i) low-order models to approximate high-dimensional dynamics; (ii) sequential data assimilation to correct the model prediction when observations become available; and (iii) an off-policy actor-critic RL algorithm to adaptively learn an optimal control strategy based on the corrected state estimates. We test DA-MIRL on the spatiotemporally chaotic solutions of the Kuramoto-Sivashinsky equation. We estimate the full state of the environment with (i) a physics-based model, here, a coarse-grained model; and (ii) a data-driven model, here, the control-aware echo state network, which is proposed in this paper. We show that DA-MIRL successfully estimates and suppresses the chaotic dynamics of the environment in real time from partial observations and approximate models. This work opens opportunities for the control of partially observable chaotic systems.

Related papers

Data-Assimilated Model-Based Reinforcement Learning for Partially Observed Chaotic Flows [3.7960472831772765]
We propose a data-assimilated model-based RL (DA-MBRL) framework for systems with partial observability and noisy measurements.<n>An off-policy actor-critic algorithm is employed to learn optimal control strategies from state estimates.<n>The framework is tested on the Kuramoto-Sivainskysh equation, demonstrating its effectiveness in stabilizing atemporally chaotic flow.
arXiv Detail & Related papers (2025-04-23T10:12:53Z)
Semi-Supervised Model-Free Bayesian State Estimation from Compressed Measurements [57.04370580292727]
We consider data-driven Bayesian state estimation from compressed measurements.<n>The dimension of the temporal measurement vector is lower than that of the temporal state vector to be estimated.<n>The underlying dynamical model of the state's evolution is unknown for a'model-free process'
arXiv Detail & Related papers (2024-07-10T05:03:48Z)
Deep autoregressive density nets vs neural ensembles for model-based offline reinforcement learning [2.9158689853305693]
We consider a model-based reinforcement learning algorithm that infers the system dynamics from the available data and performs policy optimization on imaginary model rollouts. This approach is vulnerable to exploiting model errors which can lead to catastrophic failures on the real system. We show that better performance can be obtained with a single well-calibrated autoregressive model on the D4RL benchmark.
arXiv Detail & Related papers (2024-02-05T10:18:15Z)
Data-driven Nonlinear Model Reduction using Koopman Theory: Integrated Control Form and NMPC Case Study [56.283944756315066]
We propose generic model structures combining delay-coordinate encoding of measurements and full-state decoding to integrate reduced Koopman modeling and state estimation. A case study demonstrates that our approach provides accurate control models and enables real-time capable nonlinear model predictive control of a high-purity cryogenic distillation column.
arXiv Detail & Related papers (2024-01-09T11:54:54Z)
Data Assimilation in Chaotic Systems Using Deep Reinforcement Learning [0.5999777817331317]
Data assimilation plays a pivotal role in diverse applications, ranging from climate predictions and weather forecasts to trajectory planning for autonomous vehicles. Recent advancements have seen the emergence of deep learning approaches in this domain, primarily within a supervised learning framework. In this study, we introduce a novel DA strategy that utilizes reinforcement learning (RL) to apply state corrections using full or partial observations of the state variables.
arXiv Detail & Related papers (2024-01-01T06:53:36Z)
COPlanner: Plan to Roll Out Conservatively but to Explore Optimistically for Model-Based RL [50.385005413810084]
Dyna-style model-based reinforcement learning contains two phases: model rollouts to generate sample for policy learning and real environment exploration. $textttCOPlanner$ is a planning-driven framework for model-based methods to address the inaccurately learned dynamics model problem.
arXiv Detail & Related papers (2023-10-11T06:10:07Z)
End-to-End Reinforcement Learning of Koopman Models for Economic Nonlinear Model Predictive Control [45.84205238554709]
We present a method for reinforcement learning of Koopman surrogate models for optimal performance as part of (e)NMPC. We show that the end-to-end trained models outperform those trained using system identification in (e)NMPC.
arXiv Detail & Related papers (2023-08-03T10:21:53Z)
Domain-aware Control-oriented Neural Models for Autonomous Underwater Vehicles [2.4779082385578337]
We present control-oriented parametric models with varying levels of domain-awareness. We employ universal differential equations to construct data-driven blackbox and graybox representations of the AUV dynamics.
arXiv Detail & Related papers (2022-08-15T17:01:14Z)
Dream to Explore: Adaptive Simulations for Autonomous Systems [3.0664963196464448]
We tackle the problem of learning to control dynamical systems by applying Bayesian nonparametric methods. By employing Gaussian processes to discover latent world dynamics, we mitigate common data efficiency issues observed in reinforcement learning. Our algorithm jointly learns a world model and policy by optimizing a variational lower bound of a log-likelihood.
arXiv Detail & Related papers (2021-10-27T04:27:28Z)
Using Data Assimilation to Train a Hybrid Forecast System that Combines Machine-Learning and Knowledge-Based Components [52.77024349608834]
We consider the problem of data-assisted forecasting of chaotic dynamical systems when the available data is noisy partial measurements. We show that by using partial measurements of the state of the dynamical system, we can train a machine learning model to improve predictions made by an imperfect knowledge-based model.
arXiv Detail & Related papers (2021-02-15T19:56:48Z)
Stochastically forced ensemble dynamic mode decomposition for forecasting and analysis of near-periodic systems [65.44033635330604]
We introduce a novel load forecasting method in which observed dynamics are modeled as a forced linear system. We show that its use of intrinsic linear dynamics offers a number of desirable properties in terms of interpretability and parsimony. Results are presented for a test case using load data from an electrical grid.
arXiv Detail & Related papers (2020-10-08T20:25:52Z)
Information Theoretic Model Predictive Q-Learning [64.74041985237105]
We present a novel theoretical connection between information theoretic MPC and entropy regularized RL. We develop a Q-learning algorithm that can leverage biased models.
arXiv Detail & Related papers (2019-12-31T00:29:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.