Meta-Reinforcement Learning for Adaptive Control of Second Order Systems
- URL: http://arxiv.org/abs/2209.09301v1
- Date: Mon, 19 Sep 2022 18:51:33 GMT
- Title: Meta-Reinforcement Learning for Adaptive Control of Second Order Systems
- Authors: Daniel G. McClement, Nathan P. Lawrence, Michael G. Forbes, Philip D.
Loewen, Johan U. Backstr\"om, R. Bhushan Gopaluni
- Abstract summary: In process control, many systems have similar and well-understood dynamics, which suggests it is feasible to create a generalizable controller through meta-learning.
We formulate a meta reinforcement learning (meta-RL) control strategy that takes advantage of known, offline information for training, such as a model structure.
A key design element is the ability to leverage model-based information offline during training, while maintaining a model-free policy structure for interacting with new environments.
- Score: 3.131740922192114
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Meta-learning is a branch of machine learning which aims to synthesize data
from a distribution of related tasks to efficiently solve new ones. In process
control, many systems have similar and well-understood dynamics, which suggests
it is feasible to create a generalizable controller through meta-learning. In
this work, we formulate a meta reinforcement learning (meta-RL) control
strategy that takes advantage of known, offline information for training, such
as a model structure. The meta-RL agent is trained over a distribution of model
parameters, rather than a single model, enabling the agent to automatically
adapt to changes in the process dynamics while maintaining performance. A key
design element is the ability to leverage model-based information offline
during training, while maintaining a model-free policy structure for
interacting with new environments. Our previous work has demonstrated how this
approach can be applied to the industrially-relevant problem of tuning
proportional-integral controllers to control first order processes. In this
work, we briefly reintroduce our methodology and demonstrate how it can be
extended to proportional-integral-derivative controllers and second order
systems.
Related papers
- Data-Efficient Task Generalization via Probabilistic Model-based Meta
Reinforcement Learning [58.575939354953526]
PACOH-RL is a novel model-based Meta-Reinforcement Learning (Meta-RL) algorithm designed to efficiently adapt control policies to changing dynamics.
Existing Meta-RL methods require abundant meta-learning data, limiting their applicability in settings such as robotics.
Our experiment results demonstrate that PACOH-RL outperforms model-based RL and model-based Meta-RL baselines in adapting to new dynamic conditions.
arXiv Detail & Related papers (2023-11-13T18:51:57Z) - Learning Environment Models with Continuous Stochastic Dynamics [0.0]
We aim to provide insights into the decisions faced by the agent by learning an automaton model of environmental behavior under the control of an agent.
In this work, we raise the capabilities of automata learning such that it is possible to learn models for environments that have complex and continuous dynamics.
We apply our automata learning framework on popular RL benchmarking environments in the OpenAI Gym, including LunarLander, CartPole, Mountain Car, and Acrobot.
arXiv Detail & Related papers (2023-06-29T12:47:28Z) - Unified Off-Policy Learning to Rank: a Reinforcement Learning
Perspective [61.4025671743675]
Off-policy learning to rank methods often make strong assumptions about how users generate the click data.
We show that offline reinforcement learning can adapt to various click models without complex debiasing techniques and prior knowledge of the model.
Results on various large-scale datasets demonstrate that CUOLR consistently outperforms the state-of-the-art off-policy learning to rank algorithms.
arXiv Detail & Related papers (2023-06-13T03:46:22Z) - Model-Based Reinforcement Learning with Multi-Task Offline Pretraining [59.82457030180094]
We present a model-based RL method that learns to transfer potentially useful dynamics and action demonstrations from offline data to a novel task.
The main idea is to use the world models not only as simulators for behavior learning but also as tools to measure the task relevance.
We demonstrate the advantages of our approach compared with the state-of-the-art methods in Meta-World and DeepMind Control Suite.
arXiv Detail & Related papers (2023-06-06T02:24:41Z) - A Unified Framework for Alternating Offline Model Training and Policy
Learning [62.19209005400561]
In offline model-based reinforcement learning, we learn a dynamic model from historically collected data, and utilize the learned model and fixed datasets for policy learning.
We develop an iterative offline MBRL framework, where we maximize a lower bound of the true expected return.
With the proposed unified model-policy learning framework, we achieve competitive performance on a wide range of continuous-control offline reinforcement learning datasets.
arXiv Detail & Related papers (2022-10-12T04:58:51Z) - Meta Reinforcement Learning for Adaptive Control: An Offline Approach [3.131740922192114]
We formulate a meta reinforcement learning (meta-RL) control strategy that takes advantage of known, offline information for training.
Our meta-RL agent has a recurrent structure that accumulates "context" for its current dynamics through a hidden state variable.
In tests reported here, the meta-RL agent was trained entirely offline, yet produced excellent results in novel settings.
arXiv Detail & Related papers (2022-03-17T23:58:52Z) - Learning Multi-Objective Curricula for Deep Reinforcement Learning [55.27879754113767]
Various automatic curriculum learning (ACL) methods have been proposed to improve the sample efficiency and final performance of deep reinforcement learning (DRL)
In this paper, we propose a unified automatic curriculum learning framework to create multi-objective but coherent curricula.
In addition to existing hand-designed curricula paradigms, we further design a flexible memory mechanism to learn an abstract curriculum.
arXiv Detail & Related papers (2021-10-06T19:30:25Z) - A Meta-Reinforcement Learning Approach to Process Control [3.9146761527401424]
Meta-learning aims to quickly adapt models, such as neural networks, to perform new tasks.
We construct a controller and meta-train the controller using a latent context variable through a separate embedding neural network.
In both cases, our meta-learning algorithm adapts very quickly to new tasks, outperforming a regular DRL controller trained from scratch.
arXiv Detail & Related papers (2021-03-25T18:20:56Z) - Meta Learning MPC using Finite-Dimensional Gaussian Process
Approximations [0.9539495585692008]
Two key factors that hinder the practical applicability of learning methods in control are their high computational complexity and limited generalization capabilities to unseen conditions.
This paper makes use of a meta-learning approach for adaptive model predictive control, by learning a system model that leverages data from previous related tasks.
arXiv Detail & Related papers (2020-08-13T15:59:38Z) - Information Theoretic Model Predictive Q-Learning [64.74041985237105]
We present a novel theoretical connection between information theoretic MPC and entropy regularized RL.
We develop a Q-learning algorithm that can leverage biased models.
arXiv Detail & Related papers (2019-12-31T00:29:22Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.