Learning Robust Controllers Via Probabilistic Model-Based Policy Search

• URL: http://arxiv.org/abs/2110.13576v1
• Date: Tue, 26 Oct 2021 11:17:31 GMT
• Title: Learning Robust Controllers Via Probabilistic Model-Based Policy Search
• Authors: Valentin Charvet, Bj{\o}rn Sand Jensen, Roderick Murray-Smith
• Abstract summary: We investigate whether controllers learned in such a way are robust and able to generalize under small perturbations of the environment. We show that enforcing a lower bound to the likelihood noise in the Gaussian Process dynamics model regularizes the policy updates and yields more robust controllers.
• Score: 2.886634516775814
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Model-based Reinforcement Learning estimates the true environment through a world model in order to approximate the optimal policy. This family of algorithms usually benefits from better sample efficiency than their model-free counterparts. We investigate whether controllers learned in such a way are robust and able to generalize under small perturbations of the environment. Our work is inspired by the PILCO algorithm, a method for probabilistic policy search. We show that enforcing a lower bound to the likelihood noise in the Gaussian Process dynamics model regularizes the policy updates and yields more robust controllers. We demonstrate the empirical benefits of our method in a simulation benchmark.

Related papers

• Learning Optimal Deterministic Policies with Stochastic Policy Gradients [62.81324245896716]
  Policy gradient (PG) methods are successful approaches to deal with continuous reinforcement learning (RL) problems. In common practice, convergence (hyper)policies are learned only to deploy their deterministic version. We show how to tune the exploration level used for learning to optimize the trade-off between the sample complexity and the performance of the deployed deterministic policy.
  arXiv  Detail & Related papers  (2024-05-03T16:45:15Z)
• Probabilistic Reach-Avoid for Bayesian Neural Networks [71.67052234622781]
  We show that an optimal synthesis algorithm can provide more than a four-fold increase in the number of certifiable states. The algorithm is able to provide more than a three-fold increase in the average guaranteed reach-avoid probability.
  arXiv  Detail & Related papers  (2023-10-03T10:52:21Z)
• Enabling Efficient, Reliable Real-World Reinforcement Learning with Approximate Physics-Based Models [10.472792899267365]
  We focus on developing efficient and reliable policy optimization strategies for robot learning with real-world data. In this paper we introduce a novel policy gradient-based policy optimization framework. We show that our approach can learn precise control strategies reliably and with only minutes of real-world data.
  arXiv  Detail & Related papers  (2023-07-16T22:36:36Z)
• Model-based Safe Deep Reinforcement Learning via a Constrained Proximal Policy Optimization Algorithm [4.128216503196621]
  We propose an On-policy Model-based Safe Deep RL algorithm in which we learn the transition dynamics of the environment in an online manner. We show that our algorithm is more sample efficient and results in lower cumulative hazard violations as compared to constrained model-free approaches.
  arXiv  Detail & Related papers  (2022-10-14T06:53:02Z)
• Bayesian regularization of empirical MDPs [11.3458118258705]
  We take a Bayesian perspective and regularize the objective function of the Markov decision process with prior information. We evaluate our proposed algorithms on synthetic simulations and on real-world search logs of a large scale online shopping store.
  arXiv  Detail & Related papers  (2022-08-03T22:02:50Z)
• Model Generation with Provable Coverability for Offline Reinforcement Learning [14.333861814143718]
  offline optimization with dynamics-aware policy provides a new perspective for policy learning and out-of-distribution generalization. But due to the limitation under the offline setting, the learned model could not mimic real dynamics well enough to support reliable out-of-distribution exploration. We propose an algorithm to generate models optimizing their coverage for the real dynamics.
  arXiv  Detail & Related papers  (2022-06-01T08:34:09Z)
• PC-MLP: Model-based Reinforcement Learning with Policy Cover Guided Exploration [15.173628100049129]
  This work studies a model-based algorithm for both Kernelized Regulators (KNR) and linear Markov Decision Processes (MDPs) For both models, our algorithm guarantees sample complexity and only uses access to a planning oracle. Our method can also perform reward-free exploration efficiently.
  arXiv  Detail & Related papers  (2021-07-15T15:49:30Z)
• Efficient Model-Based Reinforcement Learning through Optimistic Policy Search and Planning [93.1435980666675]
  We show how optimistic exploration can be easily combined with state-of-the-art reinforcement learning algorithms. Our experiments demonstrate that optimistic exploration significantly speeds-up learning when there are penalties on actions.
  arXiv  Detail & Related papers  (2020-06-15T18:37:38Z)
• Guided Uncertainty-Aware Policy Optimization: Combining Learning and Model-Based Strategies for Sample-Efficient Policy Learning [75.56839075060819]
  Traditional robotic approaches rely on an accurate model of the environment, a detailed description of how to perform the task, and a robust perception system to keep track of the current state. reinforcement learning approaches can operate directly from raw sensory inputs with only a reward signal to describe the task, but are extremely sample-inefficient and brittle. In this work, we combine the strengths of model-based methods with the flexibility of learning-based methods to obtain a general method that is able to overcome inaccuracies in the robotics perception/actuation pipeline.
  arXiv  Detail & Related papers  (2020-05-21T19:47:05Z)
• Model-Augmented Actor-Critic: Backpropagating through Paths [81.86992776864729]
  Current model-based reinforcement learning approaches use the model simply as a learned black-box simulator. We show how to make more effective use of the model by exploiting its differentiability.
  arXiv  Detail & Related papers  (2020-05-16T19:18:10Z)
• Information Theoretic Model Predictive Q-Learning [64.74041985237105]
  We present a novel theoretical connection between information theoretic MPC and entropy regularized RL. We develop a Q-learning algorithm that can leverage biased models.
  arXiv  Detail & Related papers  (2019-12-31T00:29:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.