Task-optimal data-driven surrogate models for eNMPC via differentiable simulation and optimization

• URL: http://arxiv.org/abs/2403.14425v2
• Date: Fri, 11 Oct 2024 15:34:06 GMT
• Title: Task-optimal data-driven surrogate models for eNMPC via differentiable simulation and optimization
• Authors: Daniel Mayfrank, Na Young Ahn, Alexander Mitsos, Manuel Dahmen,
• Abstract summary: We present a method for end-to-end learning of Koopman surrogate models for optimal performance in a specific control task. We use a training algorithm that exploits the potential differentiability of environments based on mechanistic simulation models to aid the policy optimization.
• Score: 42.72938925647165
• License:
• Abstract: We present a method for end-to-end learning of Koopman surrogate models for optimal performance in a specific control task. In contrast to previous contributions that employ standard reinforcement learning (RL) algorithms, we use a training algorithm that exploits the potential differentiability of environments based on mechanistic simulation models to aid the policy optimization. We evaluate the performance of our method by comparing it to that of other controller type and training algorithm combinations on an existing economic nonlinear model predictive control (eNMPC) case study of a continuous stirred-tank reactor (CSTR) model. Compared to the benchmark methods, our method produces similar economic performance but causes considerably fewer and less severe constraint violations. Thus, for this case study, our method outperforms the others and offers a promising path toward more performant controllers that employ dynamic surrogate models.

Related papers

• How to Fine-tune the Model: Unified Model Shift and Model Bias Policy Optimization [13.440645736306267]
  This paper develops an algorithm for model-based reinforcement learning. It unifies model shift and model bias and then formulates a fine-tuning process. It achieves state-of-the-art performance on several challenging benchmark tasks.
  arXiv  Detail & Related papers  (2023-09-22T07:27:32Z)
• When to Update Your Model: Constrained Model-based Reinforcement Learning [50.74369835934703]
  We propose a novel and general theoretical scheme for a non-decreasing performance guarantee of model-based RL (MBRL) Our follow-up derived bounds reveal the relationship between model shifts and performance improvement. A further example demonstrates that learning models from a dynamically-varying number of explorations benefit the eventual returns.
  arXiv  Detail & Related papers  (2022-10-15T17:57:43Z)
• A Reinforcement Learning-based Economic Model Predictive Control Framework for Autonomous Operation of Chemical Reactors [0.5735035463793008]
  This work presents a novel framework for integrating EMPC and RL for online model parameter estimation of a class of nonlinear systems. The major advantage of this framework is its simplicity; state-of-the-art RL algorithms and EMPC schemes can be employed with minimal modifications.
  arXiv  Detail & Related papers  (2021-05-06T13:34:30Z)
• COMBO: Conservative Offline Model-Based Policy Optimization [120.55713363569845]
  Uncertainty estimation with complex models, such as deep neural networks, can be difficult and unreliable. We develop a new model-based offline RL algorithm, COMBO, that regularizes the value function on out-of-support state-actions. We find that COMBO consistently performs as well or better as compared to prior offline model-free and model-based methods.
  arXiv  Detail & Related papers  (2021-02-16T18:50:32Z)
• Control as Hybrid Inference [62.997667081978825]
  We present an implementation of CHI which naturally mediates the balance between iterative and amortised inference. We verify the scalability of our algorithm on a continuous control benchmark, demonstrating that it outperforms strong model-free and model-based baselines.
  arXiv  Detail & Related papers  (2020-07-11T19:44:09Z)
• Variational Model-based Policy Optimization [34.80171122943031]
  Model-based reinforcement learning (RL) algorithms allow us to combine model-generated data with those collected from interaction with the real system in order to alleviate the data efficiency problem in RL. We propose an objective function as a variational lower-bound of a log-likelihood of a log-likelihood to jointly learn and improve model and policy. Our experiments on a number of continuous control tasks show that despite being more complex, our model-based (E-step) algorithm, called emactoral model-based policy optimization (VMBPO), is more sample-efficient and
  arXiv  Detail & Related papers  (2020-06-09T18:30:15Z)
• Model-Augmented Actor-Critic: Backpropagating through Paths [81.86992776864729]
  Current model-based reinforcement learning approaches use the model simply as a learned black-box simulator. We show how to make more effective use of the model by exploiting its differentiability.
  arXiv  Detail & Related papers  (2020-05-16T19:18:10Z)
• Information Theoretic Model Predictive Q-Learning [64.74041985237105]
  We present a novel theoretical connection between information theoretic MPC and entropy regularized RL. We develop a Q-learning algorithm that can leverage biased models.
  arXiv  Detail & Related papers  (2019-12-31T00:29:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.