Meta Reinforcement Learning for Adaptive Control: An Offline Approach
- URL: http://arxiv.org/abs/2203.09661v1
- Date: Thu, 17 Mar 2022 23:58:52 GMT
- Title: Meta Reinforcement Learning for Adaptive Control: An Offline Approach
- Authors: Daniel G. McClement, Nathan P. Lawrence, Johan U. Backstrom, Philip D.
Loewen, Michael G. Forbes, R. Bhushan Gopaluni
- Abstract summary: We formulate a meta reinforcement learning (meta-RL) control strategy that takes advantage of known, offline information for training.
Our meta-RL agent has a recurrent structure that accumulates "context" for its current dynamics through a hidden state variable.
In tests reported here, the meta-RL agent was trained entirely offline, yet produced excellent results in novel settings.
- Score: 3.131740922192114
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Meta-learning is a branch of machine learning which trains neural network
models to synthesize a wide variety of data in order to rapidly solve new
problems. In process control, many systems have similar and well-understood
dynamics, which suggests it is feasible to create a generalizable controller
through meta-learning. In this work, we formulate a meta reinforcement learning
(meta-RL) control strategy that takes advantage of known, offline information
for training, such as the system gain or time constant, yet efficiently
controls novel systems in a completely model-free fashion. Our meta-RL agent
has a recurrent structure that accumulates "context" for its current dynamics
through a hidden state variable. This end-to-end architecture enables the agent
to automatically adapt to changes in the process dynamics. Moreover, the same
agent can be deployed on systems with previously unseen nonlinearities and
timescales. In tests reported here, the meta-RL agent was trained entirely
offline, yet produced excellent results in novel settings. A key design element
is the ability to leverage model-based information offline during training,
while maintaining a model-free policy structure for interacting with novel
environments. To illustrate the approach, we take the actions proposed by the
meta-RL agent to be changes to gains of a proportional-integral controller,
resulting in a generalized, adaptive, closed-loop tuning strategy.
Meta-learning is a promising approach for constructing sample-efficient
intelligent controllers.
Related papers
- Autonomous Vehicle Controllers From End-to-End Differentiable Simulation [60.05963742334746]
We propose a differentiable simulator and design an analytic policy gradients (APG) approach to training AV controllers.
Our proposed framework brings the differentiable simulator into an end-to-end training loop, where gradients of environment dynamics serve as a useful prior to help the agent learn a more grounded policy.
We find significant improvements in performance and robustness to noise in the dynamics, as well as overall more intuitive human-like handling.
arXiv Detail & Related papers (2024-09-12T11:50:06Z) - Data-Efficient Task Generalization via Probabilistic Model-based Meta
Reinforcement Learning [58.575939354953526]
PACOH-RL is a novel model-based Meta-Reinforcement Learning (Meta-RL) algorithm designed to efficiently adapt control policies to changing dynamics.
Existing Meta-RL methods require abundant meta-learning data, limiting their applicability in settings such as robotics.
Our experiment results demonstrate that PACOH-RL outperforms model-based RL and model-based Meta-RL baselines in adapting to new dynamic conditions.
arXiv Detail & Related papers (2023-11-13T18:51:57Z) - Action-Quantized Offline Reinforcement Learning for Robotic Skill
Learning [68.16998247593209]
offline reinforcement learning (RL) paradigm provides recipe to convert static behavior datasets into policies that can perform better than the policy that collected the data.
In this paper, we propose an adaptive scheme for action quantization.
We show that several state-of-the-art offline RL methods such as IQL, CQL, and BRAC improve in performance on benchmarks when combined with our proposed discretization scheme.
arXiv Detail & Related papers (2023-10-18T06:07:10Z) - Model-Based Reinforcement Learning with Multi-Task Offline Pretraining [59.82457030180094]
We present a model-based RL method that learns to transfer potentially useful dynamics and action demonstrations from offline data to a novel task.
The main idea is to use the world models not only as simulators for behavior learning but also as tools to measure the task relevance.
We demonstrate the advantages of our approach compared with the state-of-the-art methods in Meta-World and DeepMind Control Suite.
arXiv Detail & Related papers (2023-06-06T02:24:41Z) - A Unified Framework for Alternating Offline Model Training and Policy
Learning [62.19209005400561]
In offline model-based reinforcement learning, we learn a dynamic model from historically collected data, and utilize the learned model and fixed datasets for policy learning.
We develop an iterative offline MBRL framework, where we maximize a lower bound of the true expected return.
With the proposed unified model-policy learning framework, we achieve competitive performance on a wide range of continuous-control offline reinforcement learning datasets.
arXiv Detail & Related papers (2022-10-12T04:58:51Z) - Meta-Reinforcement Learning for Adaptive Control of Second Order Systems [3.131740922192114]
In process control, many systems have similar and well-understood dynamics, which suggests it is feasible to create a generalizable controller through meta-learning.
We formulate a meta reinforcement learning (meta-RL) control strategy that takes advantage of known, offline information for training, such as a model structure.
A key design element is the ability to leverage model-based information offline during training, while maintaining a model-free policy structure for interacting with new environments.
arXiv Detail & Related papers (2022-09-19T18:51:33Z) - A Meta-Reinforcement Learning Approach to Process Control [3.9146761527401424]
Meta-learning aims to quickly adapt models, such as neural networks, to perform new tasks.
We construct a controller and meta-train the controller using a latent context variable through a separate embedding neural network.
In both cases, our meta-learning algorithm adapts very quickly to new tasks, outperforming a regular DRL controller trained from scratch.
arXiv Detail & Related papers (2021-03-25T18:20:56Z) - Meta Learning MPC using Finite-Dimensional Gaussian Process
Approximations [0.9539495585692008]
Two key factors that hinder the practical applicability of learning methods in control are their high computational complexity and limited generalization capabilities to unseen conditions.
This paper makes use of a meta-learning approach for adaptive model predictive control, by learning a system model that leverages data from previous related tasks.
arXiv Detail & Related papers (2020-08-13T15:59:38Z) - Information Theoretic Model Predictive Q-Learning [64.74041985237105]
We present a novel theoretical connection between information theoretic MPC and entropy regularized RL.
We develop a Q-learning algorithm that can leverage biased models.
arXiv Detail & Related papers (2019-12-31T00:29:22Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.