PC-MLP: Model-based Reinforcement Learning with Policy Cover Guided Exploration

• URL: http://arxiv.org/abs/2107.07410v1
• Date: Thu, 15 Jul 2021 15:49:30 GMT
• Title: PC-MLP: Model-based Reinforcement Learning with Policy Cover Guided Exploration
• Authors: Yuda Song, Wen Sun
• Abstract summary: This work studies a model-based algorithm for both Kernelized Regulators (KNR) and linear Markov Decision Processes (MDPs) For both models, our algorithm guarantees sample complexity and only uses access to a planning oracle. Our method can also perform reward-free exploration efficiently.
• Score: 15.173628100049129
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Model-based Reinforcement Learning (RL) is a popular learning paradigm due to its potential sample efficiency compared to model-free RL. However, existing empirical model-based RL approaches lack the ability to explore. This work studies a computationally and statistically efficient model-based algorithm for both Kernelized Nonlinear Regulators (KNR) and linear Markov Decision Processes (MDPs). For both models, our algorithm guarantees polynomial sample complexity and only uses access to a planning oracle. Experimentally, we first demonstrate the flexibility and efficacy of our algorithm on a set of exploration challenging control tasks where existing empirical model-based RL approaches completely fail. We then show that our approach retains excellent performance even in common dense reward control benchmarks that do not require heavy exploration. Finally, we demonstrate that our method can also perform reward-free exploration efficiently. Our code can be found at https://github.com/yudasong/PCMLP.

Related papers

• Getting More Juice Out of the SFT Data: Reward Learning from Human Demonstration Improves SFT for LLM Alignment [65.15914284008973]
  We propose to leverage an Inverse Reinforcement Learning (IRL) technique to simultaneously build an reward model and a policy model. We show that the proposed algorithms converge to the stationary solutions of the IRL problem. Our results indicate that it is beneficial to leverage reward learning throughout the entire alignment process.
  arXiv  Detail & Related papers  (2024-05-28T07:11:05Z)
• Active Learning for Control-Oriented Identification of Nonlinear Systems [26.231260751633307]
  We present the first finite sample analysis of an active learning algorithm suitable for a general class of nonlinear dynamics. In certain settings, the excess control cost of our algorithm achieves the optimal rate, up to logarithmic factors. We validate our approach in simulation, showcasing the advantage of active, control-oriented exploration for controlling nonlinear systems.
  arXiv  Detail & Related papers  (2024-04-13T15:40:39Z)
• Model-Based RL for Mean-Field Games is not Statistically Harder than Single-Agent RL [57.745700271150454]
  We study the sample complexity of reinforcement learning in Mean-Field Games (MFGs) with model-based function approximation. We introduce the Partial Model-Based Eluder Dimension (P-MBED), a more effective notion to characterize the model class complexity.
  arXiv  Detail & Related papers  (2024-02-08T14:54:47Z)
• Maximize to Explore: One Objective Function Fusing Estimation, Planning, and Exploration [87.53543137162488]
  We propose an easy-to-implement online reinforcement learning (online RL) framework called textttMEX. textttMEX integrates estimation and planning components while balancing exploration exploitation automatically. It can outperform baselines by a stable margin in various MuJoCo environments with sparse rewards.
  arXiv  Detail & Related papers  (2023-05-29T17:25:26Z)
• Reinforcement Learning with Partial Parametric Model Knowledge [3.3598755777055374]
  We adapt reinforcement learning methods for continuous control to bridge the gap between complete ignorance and perfect knowledge of the environment. Our method, Partial Knowledge Least Squares Policy Iteration (PLSPI), takes inspiration from both model-free RL and model-based control.
  arXiv  Detail & Related papers  (2023-04-26T01:04:35Z)
• MoDem: Accelerating Visual Model-Based Reinforcement Learning with Demonstrations [36.44386146801296]
  Poor sample efficiency continues to be the primary challenge for deployment of deep Reinforcement Learning (RL) algorithms for real-world applications. We find that leveraging just a handful of demonstrations can dramatically improve the sample-efficiency of model-based RL. We empirically study three complex visuo-motor control domains and find that our method is 150%-250% more successful in completing sparse reward tasks.
  arXiv  Detail & Related papers  (2022-12-12T04:28:50Z)
• CostNet: An End-to-End Framework for Goal-Directed Reinforcement Learning [9.432068833600884]
  Reinforcement Learning (RL) is a general framework concerned with an agent that seeks to maximize rewards in an environment. There are two approaches, model-based and model-free reinforcement learning, that show concrete results in several disciplines. This paper introduces a novel reinforcement learning algorithm for predicting the distance between two states in a Markov Decision Process.
  arXiv  Detail & Related papers  (2022-10-03T21:16:14Z)
• Simplifying Model-based RL: Learning Representations, Latent-space Models, and Policies with One Objective [142.36200080384145]
  We propose a single objective which jointly optimize a latent-space model and policy to achieve high returns while remaining self-consistent. We demonstrate that the resulting algorithm matches or improves the sample-efficiency of the best prior model-based and model-free RL methods.
  arXiv  Detail & Related papers  (2022-09-18T03:51:58Z)
• Learning Robust Controllers Via Probabilistic Model-Based Policy Search [2.886634516775814]
  We investigate whether controllers learned in such a way are robust and able to generalize under small perturbations of the environment. We show that enforcing a lower bound to the likelihood noise in the Gaussian Process dynamics model regularizes the policy updates and yields more robust controllers.
  arXiv  Detail & Related papers  (2021-10-26T11:17:31Z)
• Efficient Model-Based Reinforcement Learning through Optimistic Policy Search and Planning [93.1435980666675]
  We show how optimistic exploration can be easily combined with state-of-the-art reinforcement learning algorithms. Our experiments demonstrate that optimistic exploration significantly speeds-up learning when there are penalties on actions.
  arXiv  Detail & Related papers  (2020-06-15T18:37:38Z)
• Information Theoretic Model Predictive Q-Learning [64.74041985237105]
  We present a novel theoretical connection between information theoretic MPC and entropy regularized RL. We develop a Q-learning algorithm that can leverage biased models.
  arXiv  Detail & Related papers  (2019-12-31T00:29:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.