High-Accuracy Model-Based Reinforcement Learning, a Survey

• URL: http://arxiv.org/abs/2107.08241v1
• Date: Sat, 17 Jul 2021 14:01:05 GMT
• Title: High-Accuracy Model-Based Reinforcement Learning, a Survey
• Authors: Aske Plaat and Walter Kosters and Mike Preuss
• Abstract summary: Deep reinforcement learning has shown remarkable success in game playing and robotics. To reduce the number of environment samples, model-based reinforcement learning creates an explicit model of the environment dynamics. Some of these methods succeed in achieving high accuracy at low sample complexity, most do so either in a robotics or in a games context.
• Score: 2.0196229393131726
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Deep reinforcement learning has shown remarkable success in the past few years. Highly complex sequential decision making problems from game playing and robotics have been solved with deep model-free methods. Unfortunately, the sample complexity of model-free methods is often high. To reduce the number of environment samples, model-based reinforcement learning creates an explicit model of the environment dynamics. Achieving high model accuracy is a challenge in high-dimensional problems. In recent years, a diverse landscape of model-based methods has been introduced to improve model accuracy, using methods such as uncertainty modeling, model-predictive control, latent models, and end-to-end learning and planning. Some of these methods succeed in achieving high accuracy at low sample complexity, most do so either in a robotics or in a games context. In this paper, we survey these methods; we explain in detail how they work and what their strengths and weaknesses are. We conclude with a research agenda for future work to make the methods more robust and more widely applicable to other applications.

Related papers

• Learning-based Models for Vulnerability Detection: An Extensive Study [3.1317409221921144]
  We extensively and comprehensively investigate two types of state-of-the-art learning-based approaches. We experimentally demonstrate the priority of sequence-based models and the limited abilities of both graph-based models.
  arXiv  Detail & Related papers  (2024-08-14T13:01:30Z)
• Towards Learning Stochastic Population Models by Gradient Descent [0.0]
  We show that simultaneous estimation of parameters and structure poses major challenges for optimization procedures. We demonstrate accurate estimation of models but find that enforcing the inference of parsimonious, interpretable models drastically increases the difficulty.
  arXiv  Detail & Related papers  (2024-04-10T14:38:58Z)
• Fine-Tuning Generative Models as an Inference Method for Robotic Tasks [18.745665662647912]
  We investigate how to quickly adapt the sample generation of neural network models to observations in robotic tasks. The key idea is to quickly fine-tune the model by fitting it to generated samples matching the observed evidence. We show that our method can be applied to both autoregressive models and variational autoencoders.
  arXiv  Detail & Related papers  (2023-10-19T16:11:49Z)
• Minimal Value-Equivalent Partial Models for Scalable and Robust Planning in Lifelong Reinforcement Learning [56.50123642237106]
  Common practice in model-based reinforcement learning is to learn models that model every aspect of the agent's environment. We argue that such models are not particularly well-suited for performing scalable and robust planning in lifelong reinforcement learning scenarios. We propose new kinds of models that only model the relevant aspects of the environment, which we call "minimal value-minimal partial models"
  arXiv  Detail & Related papers  (2023-01-24T16:40:01Z)
• Sample-Efficient Reinforcement Learning via Conservative Model-Based Actor-Critic [67.00475077281212]
  Model-based reinforcement learning algorithms are more sample efficient than their model-free counterparts. We propose a novel approach that achieves high sample efficiency without the strong reliance on accurate learned models. We show that CMBAC significantly outperforms state-of-the-art approaches in terms of sample efficiency on several challenging tasks.
  arXiv  Detail & Related papers  (2021-12-16T15:33:11Z)
• Sample Efficient Reinforcement Learning via Model-Ensemble Exploration and Exploitation [3.728946517493471]
  MEEE is a model-ensemble method that consists of optimistic exploration and weighted exploitation. Our approach outperforms other model-free and model-based state-of-the-art methods, especially in sample complexity.
  arXiv  Detail & Related papers  (2021-07-05T07:18:20Z)
• Sufficiently Accurate Model Learning for Planning [119.80502738709937]
  This paper introduces the constrained Sufficiently Accurate model learning approach. It provides examples of such problems, and presents a theorem on how close some approximate solutions can be. The approximate solution quality will depend on the function parameterization, loss and constraint function smoothness, and the number of samples in model learning.
  arXiv  Detail & Related papers  (2021-02-11T16:27:31Z)
• Model-Based Visual Planning with Self-Supervised Functional Distances [104.83979811803466]
  We present a self-supervised method for model-based visual goal reaching. Our approach learns entirely using offline, unlabeled data. We find that this approach substantially outperforms both model-free and model-based prior methods.
  arXiv  Detail & Related papers  (2020-12-30T23:59:09Z)
• Deep Model-Based Reinforcement Learning for High-Dimensional Problems, a Survey [1.2031796234206134]
  Model-based reinforcement learning creates an explicit model of the environment dynamics to reduce the need for environment samples. A challenge for deep model-based methods is to achieve high predictive power while maintaining low sample complexity. We propose a taxonomy based on three approaches: using explicit planning on given transitions, using explicit planning on learned transitions, and end-to-end learning of both planning and transitions.
  arXiv  Detail & Related papers  (2020-08-11T08:49:04Z)
• Goal-Aware Prediction: Learning to Model What Matters [105.43098326577434]
  One of the fundamental challenges in using a learned forward dynamics model is the mismatch between the objective of the learned model and that of the downstream planner or policy. We propose to direct prediction towards task relevant information, enabling the model to be aware of the current task and encouraging it to only model relevant quantities of the state space. We find that our method more effectively models the relevant parts of the scene conditioned on the goal, and as a result outperforms standard task-agnostic dynamics models and model-free reinforcement learning.
  arXiv  Detail & Related papers  (2020-07-14T16:42:59Z)
• Hybrid modeling: Applications in real-time diagnosis [64.5040763067757]
  We outline a novel hybrid modeling approach that combines machine learning inspired models and physics-based models. We are using such models for real-time diagnosis applications.
  arXiv  Detail & Related papers  (2020-03-04T00:44:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.