Model-based versus Model-free Deep Reinforcement Learning for Autonomous
Racing Cars
- URL: http://arxiv.org/abs/2103.04909v1
- Date: Mon, 8 Mar 2021 17:15:23 GMT
- Title: Model-based versus Model-free Deep Reinforcement Learning for Autonomous
Racing Cars
- Authors: Axel Brunnbauer, Luigi Berducci, Andreas Brandst\"atter, Mathias
Lechner, Ramin Hasani, Daniela Rus, Radu Grosu
- Abstract summary: This paper investigates how model-based deep reinforcement learning agents generalize to real-world autonomous-vehicle control-tasks.
We show that model-based agents capable of learning in imagination, substantially outperform model-free agents with respect to performance, sample efficiency, successful task completion, and generalization.
- Score: 46.64253693115981
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Despite the rich theoretical foundation of model-based deep reinforcement
learning (RL) agents, their effectiveness in real-world robotics-applications
is less studied and understood. In this paper, we, therefore, investigate how
such agents generalize to real-world autonomous-vehicle control-tasks, where
advanced model-free deep RL algorithms fail. In particular, we set up a series
of time-lap tasks for an F1TENTH racing robot, equipped with high-dimensional
LiDAR sensors, on a set of test tracks with a gradual increase in their
complexity. In this continuous-control setting, we show that model-based agents
capable of learning in imagination, substantially outperform model-free agents
with respect to performance, sample efficiency, successful task completion, and
generalization. Moreover, we show that the generalization ability of
model-based agents strongly depends on the observation-model choice. Finally,
we provide extensive empirical evidence for the effectiveness of model-based
agents provided with long enough memory horizons in sim2real tasks.
Related papers
- Active Exploration in Bayesian Model-based Reinforcement Learning for Robot Manipulation [8.940998315746684]
We propose a model-based reinforcement learning (RL) approach for robotic arm end-tasks.
We employ Bayesian neural network models to represent, in a probabilistic way, both the belief and information encoded in the dynamic model during exploration.
Our experiments show the advantages of our Bayesian model-based RL approach, with similar quality in the results than relevant alternatives.
arXiv Detail & Related papers (2024-04-02T11:44:37Z) - LORD: Large Models based Opposite Reward Design for Autonomous Driving [11.717821043996352]
We introduce LORD, a novel large models based opposite reward design through undesired linguistic goals.
Our proposed framework shows its efficiency in leveraging the power of large pretrained models for achieving safe and enhanced autonomous driving.
arXiv Detail & Related papers (2024-03-27T19:30:06Z) - Simplified Temporal Consistency Reinforcement Learning [19.814047499837084]
We show that a simple representation learning approach relying on a latent dynamics model trained by latent temporal consistency is sufficient for high-performance RL.
Our approach outperforms model-free methods by a large margin and matches model-based methods' sample efficiency while training 2.4 times faster.
arXiv Detail & Related papers (2023-06-15T19:37:43Z) - Sense, Imagine, Act: Multimodal Perception Improves Model-Based
Reinforcement Learning for Head-to-Head Autonomous Racing [10.309579267966361]
Model-based reinforcement learning (MBRL) techniques have recently yielded promising results for real-world autonomous racing.
This paper proposes a self-supervised sensor fusion technique that combines egocentric LiDAR and RGB camera observations collected from the F1TENTH Gym.
The resulting Dreamer agent safely avoided collisions and won the most races compared to other tested baselines in zero-shot head-to-head autonomous racing.
arXiv Detail & Related papers (2023-05-08T14:49:02Z) - Learning a model is paramount for sample efficiency in reinforcement
learning control of PDEs [5.488334211013093]
We show that learning an actuated model in parallel to training the RL agent significantly reduces the total amount of required data sampled from the real system.
We also show that iteratively updating the model is of major importance to avoid biases in the RL training.
arXiv Detail & Related papers (2023-02-14T16:14:39Z) - Simplifying Model-based RL: Learning Representations, Latent-space
Models, and Policies with One Objective [142.36200080384145]
We propose a single objective which jointly optimize a latent-space model and policy to achieve high returns while remaining self-consistent.
We demonstrate that the resulting algorithm matches or improves the sample-efficiency of the best prior model-based and model-free RL methods.
arXiv Detail & Related papers (2022-09-18T03:51:58Z) - INFOrmation Prioritization through EmPOWERment in Visual Model-Based RL [90.06845886194235]
We propose a modified objective for model-based reinforcement learning (RL)
We integrate a term inspired by variational empowerment into a state-space model based on mutual information.
We evaluate the approach on a suite of vision-based robot control tasks with natural video backgrounds.
arXiv Detail & Related papers (2022-04-18T23:09:23Z) - Multitask Adaptation by Retrospective Exploration with Learned World
Models [77.34726150561087]
We propose a meta-learned addressing model called RAMa that provides training samples for the MBRL agent taken from task-agnostic storage.
The model is trained to maximize the expected agent's performance by selecting promising trajectories solving prior tasks from the storage.
arXiv Detail & Related papers (2021-10-25T20:02:57Z) - Online reinforcement learning with sparse rewards through an active
inference capsule [62.997667081978825]
This paper introduces an active inference agent which minimizes the novel free energy of the expected future.
Our model is capable of solving sparse-reward problems with a very high sample efficiency.
We also introduce a novel method for approximating the prior model from the reward function, which simplifies the expression of complex objectives.
arXiv Detail & Related papers (2021-06-04T10:03:36Z) - Goal-Aware Prediction: Learning to Model What Matters [105.43098326577434]
One of the fundamental challenges in using a learned forward dynamics model is the mismatch between the objective of the learned model and that of the downstream planner or policy.
We propose to direct prediction towards task relevant information, enabling the model to be aware of the current task and encouraging it to only model relevant quantities of the state space.
We find that our method more effectively models the relevant parts of the scene conditioned on the goal, and as a result outperforms standard task-agnostic dynamics models and model-free reinforcement learning.
arXiv Detail & Related papers (2020-07-14T16:42:59Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.