Uncertainty-Aware Model-Based Reinforcement Learning with Application to
Autonomous Driving
- URL: http://arxiv.org/abs/2106.12194v1
- Date: Wed, 23 Jun 2021 06:55:14 GMT
- Title: Uncertainty-Aware Model-Based Reinforcement Learning with Application to
Autonomous Driving
- Authors: Jingda Wu, Zhiyu Huang, Chen Lv
- Abstract summary: We propose a novel uncertainty-aware model-based reinforcement learning framework, and then implement and validate it in autonomous driving.
The framework is developed based on the adaptive truncation approach, providing virtual interactions between the agent and environment model.
The developed algorithms are then implemented in end-to-end autonomous vehicle control tasks, validated and compared with state-of-the-art methods under various driving scenarios.
- Score: 2.3303341607459687
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: To further improve the learning efficiency and performance of reinforcement
learning (RL), in this paper we propose a novel uncertainty-aware model-based
RL (UA-MBRL) framework, and then implement and validate it in autonomous
driving under various task scenarios. First, an action-conditioned ensemble
model with the ability of uncertainty assessment is established as the virtual
environment model. Then, a novel uncertainty-aware model-based RL framework is
developed based on the adaptive truncation approach, providing virtual
interactions between the agent and environment model, and improving RL's
training efficiency and performance. The developed algorithms are then
implemented in end-to-end autonomous vehicle control tasks, validated and
compared with state-of-the-art methods under various driving scenarios. The
validation results suggest that the proposed UA-MBRL method surpasses the
existing model-based and model-free RL approaches, in terms of learning
efficiency and achieved performance. The results also demonstrate the good
ability of the proposed method with respect to the adaptiveness and robustness,
under various autonomous driving scenarios.
Related papers
- Safe Deep Model-Based Reinforcement Learning with Lyapunov Functions [2.50194939587674]
We propose a new Model-based RL framework to enable efficient policy learning with unknown dynamics.
We introduce and explore a novel method for adding safety constraints for model-based RL during training and policy learning.
arXiv Detail & Related papers (2024-05-25T11:21:12Z) - Simplified Temporal Consistency Reinforcement Learning [19.814047499837084]
We show that a simple representation learning approach relying on a latent dynamics model trained by latent temporal consistency is sufficient for high-performance RL.
Our approach outperforms model-free methods by a large margin and matches model-based methods' sample efficiency while training 2.4 times faster.
arXiv Detail & Related papers (2023-06-15T19:37:43Z) - Latent Variable Representation for Reinforcement Learning [131.03944557979725]
It remains unclear theoretically and empirically how latent variable models may facilitate learning, planning, and exploration to improve the sample efficiency of model-based reinforcement learning.
We provide a representation view of the latent variable models for state-action value functions, which allows both tractable variational learning algorithm and effective implementation of the optimism/pessimism principle.
In particular, we propose a computationally efficient planning algorithm with UCB exploration by incorporating kernel embeddings of latent variable models.
arXiv Detail & Related papers (2022-12-17T00:26:31Z) - Active Learning of Discrete-Time Dynamics for Uncertainty-Aware Model
Predictive Control [49.60520501097199]
We present a self-supervised learning approach that actively models the dynamics of nonlinear robotic systems.
Our approach showcases high resilience and generalization capabilities by consistently adapting to unseen flight conditions.
arXiv Detail & Related papers (2022-10-23T00:45:05Z) - When to Update Your Model: Constrained Model-based Reinforcement
Learning [50.74369835934703]
We propose a novel and general theoretical scheme for a non-decreasing performance guarantee of model-based RL (MBRL)
Our follow-up derived bounds reveal the relationship between model shifts and performance improvement.
A further example demonstrates that learning models from a dynamically-varying number of explorations benefit the eventual returns.
arXiv Detail & Related papers (2022-10-15T17:57:43Z) - UMBRELLA: Uncertainty-Aware Model-Based Offline Reinforcement Learning
Leveraging Planning [1.1339580074756188]
Offline reinforcement learning (RL) provides a framework for learning decision-making from offline data.
Self-driving vehicles (SDV) learn a policy, which potentially even outperforms the behavior in the sub-optimal data set.
This motivates the use of model-based offline RL approaches, which leverage planning.
arXiv Detail & Related papers (2021-11-22T10:37:52Z) - A Reinforcement Learning-based Economic Model Predictive Control
Framework for Autonomous Operation of Chemical Reactors [0.5735035463793008]
This work presents a novel framework for integrating EMPC and RL for online model parameter estimation of a class of nonlinear systems.
The major advantage of this framework is its simplicity; state-of-the-art RL algorithms and EMPC schemes can be employed with minimal modifications.
arXiv Detail & Related papers (2021-05-06T13:34:30Z) - COMBO: Conservative Offline Model-Based Policy Optimization [120.55713363569845]
Uncertainty estimation with complex models, such as deep neural networks, can be difficult and unreliable.
We develop a new model-based offline RL algorithm, COMBO, that regularizes the value function on out-of-support state-actions.
We find that COMBO consistently performs as well or better as compared to prior offline model-free and model-based methods.
arXiv Detail & Related papers (2021-02-16T18:50:32Z) - Control as Hybrid Inference [62.997667081978825]
We present an implementation of CHI which naturally mediates the balance between iterative and amortised inference.
We verify the scalability of our algorithm on a continuous control benchmark, demonstrating that it outperforms strong model-free and model-based baselines.
arXiv Detail & Related papers (2020-07-11T19:44:09Z) - Information Theoretic Model Predictive Q-Learning [64.74041985237105]
We present a novel theoretical connection between information theoretic MPC and entropy regularized RL.
We develop a Q-learning algorithm that can leverage biased models.
arXiv Detail & Related papers (2019-12-31T00:29:22Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.