Robust Model Based Reinforcement Learning Using $\mathcal{L}_1$ Adaptive Control

• URL: http://arxiv.org/abs/2403.14860v1
• Date: Thu, 21 Mar 2024 22:15:09 GMT
• Title: Robust Model Based Reinforcement Learning Using $\mathcal{L}_1$ Adaptive Control
• Authors: Minjun Sung, Sambhu H. Karumanchi, Aditya Gahlawat, Naira Hovakimyan,
• Abstract summary: We introduce a control-theoretic augmentation scheme for Model-Based Reinforcement Learning (MBRL) algorithms. MBRL algorithms learn a model of the transition function using data and use it to design a control input. Our approach generates a series of approximate control-affine models of the learned transition function according to the proposed switching law.
• Score: 4.88489286130994
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce $\mathcal{L}_1$-MBRL, a control-theoretic augmentation scheme for Model-Based Reinforcement Learning (MBRL) algorithms. Unlike model-free approaches, MBRL algorithms learn a model of the transition function using data and use it to design a control input. Our approach generates a series of approximate control-affine models of the learned transition function according to the proposed switching law. Using the approximate model, control input produced by the underlying MBRL is perturbed by the $\mathcal{L}_1$ adaptive control, which is designed to enhance the robustness of the system against uncertainties. Importantly, this approach is agnostic to the choice of MBRL algorithm, enabling the use of the scheme with various MBRL algorithms. MBRL algorithms with $\mathcal{L}_1$ augmentation exhibit enhanced performance and sample efficiency across multiple MuJoCo environments, outperforming the original MBRL algorithms, both with and without system noise.

Related papers

• Bayes Adaptive Monte Carlo Tree Search for Offline Model-based Reinforcement Learning [5.663006149337036]
  offline model-based reinforcement learning (MBRL) is a powerful approach for data-driven decision-making and control. There could be various MDPs that behave identically on the offline dataset and so dealing with the uncertainty about the true MDP can be challenging. We introduce a novel Bayes Adaptive Monte-Carlo planning algorithm capable of solving BAMDPs in continuous state and action spaces.
  arXiv  Detail & Related papers  (2024-10-15T03:36:43Z)
• Understanding Reinforcement Learning-Based Fine-Tuning of Diffusion Models: A Tutorial and Review [63.31328039424469]
  This tutorial provides a comprehensive survey of methods for fine-tuning diffusion models to optimize downstream reward functions. We explain the application of various RL algorithms, including PPO, differentiable optimization, reward-weighted MLE, value-weighted sampling, and path consistency learning.
  arXiv  Detail & Related papers  (2024-07-18T17:35:32Z)
• Adding Conditional Control to Diffusion Models with Reinforcement Learning [59.295203871547336]
  Diffusion models are powerful generative models that allow for precise control over the characteristics of the generated samples. This work presents a novel method based on reinforcement learning (RL) to add additional controls, leveraging an offline dataset.
  arXiv  Detail & Related papers  (2024-06-17T22:00:26Z)
• Efficient model predictive control for nonlinear systems modelled by deep neural networks [6.5268245109828005]
  This paper presents a model predictive control (MPC) for dynamic systems whose nonlinearity and uncertainty are modelled by deep neural networks (NNs) Since the NN output contains a high-order complex nonlinearity of the system state and control input, the MPC problem is nonlinear and challenging to solve for real-time control.
  arXiv  Detail & Related papers  (2024-05-16T18:05:18Z)
• Oracle Inequalities for Model Selection in Offline Reinforcement Learning [105.74139523696284]
  We study the problem of model selection in offline RL with value function approximation. We propose the first model selection algorithm for offline RL that achieves minimax rate-optimal inequalities up to logarithmic factors. We conclude with several numerical simulations showing it is capable of reliably selecting a good model class.
  arXiv  Detail & Related papers  (2022-11-03T17:32:34Z)
• When to Update Your Model: Constrained Model-based Reinforcement Learning [50.74369835934703]
  We propose a novel and general theoretical scheme for a non-decreasing performance guarantee of model-based RL (MBRL) Our follow-up derived bounds reveal the relationship between model shifts and performance improvement. A further example demonstrates that learning models from a dynamically-varying number of explorations benefit the eventual returns.
  arXiv  Detail & Related papers  (2022-10-15T17:57:43Z)
• An Analysis of Model-Based Reinforcement Learning From Abstracted Observations [24.964038353043918]
  We show that abstraction can introduce a dependence between samples collected online (e.g., in the real world) and results for Model-based Reinforcement learning (MBRL) We show that we can use concentration inequalities for martingales to overcome this problem. We illustrate this by combining R-MAX, a prototypical MBRL algorithm, with abstraction, thus producing the first performance guarantees for model-based 'RL from Abstracted Observations'
  arXiv  Detail & Related papers  (2022-08-30T17:19:26Z)
• Robust optimal well control using an adaptive multi-grid reinforcement learning framework [0.0]
  Reinforcement learning is a promising tool to solve robust optimal well control problems. The proposed framework is demonstrated using a state-of-the-art, model-free policy-based RL algorithm. Prominent gains in the computational efficiency is observed using the proposed framework saving around 60-70% of computational cost of its single fine-grid counterpart.
  arXiv  Detail & Related papers  (2022-07-07T12:08:57Z)
• Exponential Family Model-Based Reinforcement Learning via Score Matching [97.31477125728844]
  We propose an optimistic model-based algorithm, dubbed SMRL, for finitehorizon episodic reinforcement learning (RL) SMRL uses score matching, an unnormalized density estimation technique that enables efficient estimation of the model parameter by ridge regression.
  arXiv  Detail & Related papers  (2021-12-28T15:51:07Z)
• Policy Search using Dynamic Mirror Descent MPC for Model Free Off Policy RL [0.0]
  Recent works in Reinforcement Learning (RL) combine model-free (Mf)-RL algorithms with model-based (Mb)-RL approaches. We propose a hierarchical framework that integrates online learning for the Mb-trajectory optimization with off-policy methods for the Mf-RL.
  arXiv  Detail & Related papers  (2021-10-23T15:16:49Z)
• Information Theoretic Model Predictive Q-Learning [64.74041985237105]
  We present a novel theoretical connection between information theoretic MPC and entropy regularized RL. We develop a Q-learning algorithm that can leverage biased models.
  arXiv  Detail & Related papers  (2019-12-31T00:29:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.