Data-driven control of spatiotemporal chaos with reduced-order neural ODE-based models and reinforcement learning

• URL: http://arxiv.org/abs/2205.00579v1
• Date: Sun, 1 May 2022 23:25:44 GMT
• Title: Data-driven control of spatiotemporal chaos with reduced-order neural ODE-based models and reinforcement learning
• Authors: Kevin Zeng, Alec J. Linot, Michael D. Graham
• Abstract summary: Deep learning is capable of discovering complex control strategies for high-dimensional systems, making it promising for flow control applications. A major challenge associated with RL is that substantial training data must be generated by repeatedly interacting with the target system. We use a data-driven reduced-order model (ROM) in place the true system during RL training to efficiently estimate the optimal policy. We show that the ROM-based control strategy translates well to the true KSE and highlight that the RL agent discovers and stabilizes an underlying forced equilibrium solution of the KSE system.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Deep reinforcement learning (RL) is a data-driven method capable of discovering complex control strategies for high-dimensional systems, making it promising for flow control applications. In particular, the present work is motivated by the goal of reducing energy dissipation in turbulent flows, and the example considered is the spatiotemporally chaotic dynamics of the Kuramoto-Sivashinsky equation (KSE). A major challenge associated with RL is that substantial training data must be generated by repeatedly interacting with the target system, making it costly when the system is computationally or experimentally expensive. We mitigate this challenge in a data-driven manner by combining dimensionality reduction via an autoencoder with a neural ODE framework to obtain a low-dimensional dynamical model from just a limited data set. We substitute this data-driven reduced-order model (ROM) in place of the true system during RL training to efficiently estimate the optimal policy, which can then be deployed on the true system. For the KSE actuated with localized forcing ("jets") at four locations, we demonstrate that we are able to learn a ROM that accurately captures the actuated dynamics as well as the underlying natural dynamics just from snapshots of the KSE experiencing random actuations. Using this ROM and a control objective of minimizing dissipation and power cost, we extract a control policy from it using deep RL. We show that the ROM-based control strategy translates well to the true KSE and highlight that the RL agent discovers and stabilizes an underlying forced equilibrium solution of the KSE system. We show that this forced equilibrium captured in the ROM and discovered through RL is related to an existing known equilibrium solution of the natural KSE.

Related papers

• Neural Internal Model Control: Learning a Robust Control Policy via Predictive Error Feedback [16.46487826869775]
  We propose a novel framework, Neural Internal Model Control, which integrates model-based control with RL-based control to enhance robustness. Our framework streamlines the predictive model by applying Newton-Euler equations for rigid-body dynamics, eliminating the need to capture complex high-dimensional nonlinearities. We demonstrate the effectiveness of our framework on both quadrotors and quadrupedal robots, achieving superior performance compared to state-of-the-art methods.
  arXiv  Detail & Related papers  (2024-11-20T07:07:42Z)
• SINDy-RL: Interpretable and Efficient Model-Based Reinforcement Learning [5.59265003686955]
  We introduce SINDy-RL, a framework for combining SINDy and deep reinforcement learning. SINDy-RL achieves comparable performance to state-of-the-art DRL algorithms. We demonstrate the effectiveness of our approaches on benchmark control environments and challenging fluids problems.
  arXiv  Detail & Related papers  (2024-03-14T05:17:39Z)
• Compressing Deep Reinforcement Learning Networks with a Dynamic Structured Pruning Method for Autonomous Driving [63.155562267383864]
  Deep reinforcement learning (DRL) has shown remarkable success in complex autonomous driving scenarios. DRL models inevitably bring high memory consumption and computation, which hinders their wide deployment in resource-limited autonomous driving devices. We introduce a novel dynamic structured pruning approach that gradually removes a DRL model's unimportant neurons during the training stage.
  arXiv  Detail & Related papers  (2024-02-07T09:00:30Z)
• Data-Efficient Task Generalization via Probabilistic Model-based Meta Reinforcement Learning [58.575939354953526]
  PACOH-RL is a novel model-based Meta-Reinforcement Learning (Meta-RL) algorithm designed to efficiently adapt control policies to changing dynamics. Existing Meta-RL methods require abundant meta-learning data, limiting their applicability in settings such as robotics. Our experiment results demonstrate that PACOH-RL outperforms model-based RL and model-based Meta-RL baselines in adapting to new dynamic conditions.
  arXiv  Detail & Related papers  (2023-11-13T18:51:57Z)
• Learning a model is paramount for sample efficiency in reinforcement learning control of PDEs [5.488334211013093]
  We show that learning an actuated model in parallel to training the RL agent significantly reduces the total amount of required data sampled from the real system. We also show that iteratively updating the model is of major importance to avoid biases in the RL training.
  arXiv  Detail & Related papers  (2023-02-14T16:14:39Z)
• Efficient Learning of Voltage Control Strategies via Model-based Deep Reinforcement Learning [9.936452412191326]
  This article proposes a model-based deep reinforcement learning (DRL) method to design emergency control strategies for short-term voltage stability problems in power systems. Recent advances show promising results in model-free DRL-based methods for power systems, but model-free methods suffer from poor sample efficiency and training time. We propose a novel model-based-DRL framework where a deep neural network (DNN)-based dynamic surrogate model is utilized with the policy learning framework.
  arXiv  Detail & Related papers  (2022-12-06T02:50:53Z)
• Symmetry reduction for deep reinforcement learning active control of chaotic spatiotemporal dynamics [0.0]
  Deep reinforcement learning (RL) is capable of discovering complex control strategies for macroscopic objectives in high-dimensional systems. We show that by moving the deep RL problem to a symmetry-reduced space, we can alleviate limitations inherent in the naive application of deep RL. We demonstrate that symmetry-reduced deep RL yields improved data efficiency as well as improved control policy efficacy compared to policies found by naive deep RL.
  arXiv  Detail & Related papers  (2021-04-09T17:55:12Z)
• An Ode to an ODE [78.97367880223254]
  We present a new paradigm for Neural ODE algorithms, called ODEtoODE, where time-dependent parameters of the main flow evolve according to a matrix flow on the group O(d) This nested system of two flows provides stability and effectiveness of training and provably solves the gradient vanishing-explosion problem.
  arXiv  Detail & Related papers  (2020-06-19T22:05:19Z)
• Online Reinforcement Learning Control by Direct Heuristic Dynamic Programming: from Time-Driven to Event-Driven [80.94390916562179]
  Time-driven learning refers to the machine learning method that updates parameters in a prediction model continuously as new data arrives. It is desirable to prevent the time-driven dHDP from updating due to insignificant system event such as noise. We show how the event-driven dHDP algorithm works in comparison to the original time-driven dHDP.
  arXiv  Detail & Related papers  (2020-06-16T05:51:25Z)
• Adaptive Control and Regret Minimization in Linear Quadratic Gaussian (LQG) Setting [91.43582419264763]
  We propose LqgOpt, a novel reinforcement learning algorithm based on the principle of optimism in the face of uncertainty. LqgOpt efficiently explores the system dynamics, estimates the model parameters up to their confidence interval, and deploys the controller of the most optimistic model.
  arXiv  Detail & Related papers  (2020-03-12T19:56:38Z)
• Information Theoretic Model Predictive Q-Learning [64.74041985237105]
  We present a novel theoretical connection between information theoretic MPC and entropy regularized RL. We develop a Q-learning algorithm that can leverage biased models.
  arXiv  Detail & Related papers  (2019-12-31T00:29:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.