Inferring Smooth Control: Monte Carlo Posterior Policy Iteration with Gaussian Processes

• URL: http://arxiv.org/abs/2210.03512v1
• Date: Fri, 7 Oct 2022 12:56:31 GMT
• Title: Inferring Smooth Control: Monte Carlo Posterior Policy Iteration with Gaussian Processes
• Authors: Joe Watson and Jan Peters
• Abstract summary: We show how to achieve smoother model predictive factor control using online sequential inference. We evaluate this approach on several robot control tasks, matching to sample prior methods while also ensuring smoothness.
• Score: 39.411957858548355
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Monte Carlo methods have become increasingly relevant for control of non-differentiable systems, approximate dynamics models and learning from data. These methods scale to high-dimensional spaces and are effective at the non-convex optimizations often seen in robot learning. We look at sample-based methods from the perspective of inference-based control, specifically posterior policy iteration. From this perspective, we highlight how Gaussian noise priors produce rough control actions that are unsuitable for physical robot deployment. Considering smoother Gaussian process priors, as used in episodic reinforcement learning and motion planning, we demonstrate how smoother model predictive control can be achieved using online sequential inference. This inference is realized through an efficient factorization of the action distribution and a novel means of optimizing the likelihood temperature to improve importance sampling accuracy. We evaluate this approach on several high-dimensional robot control tasks, matching the sample efficiency of prior heuristic methods while also ensuring smoothness. Simulation results can be seen at https://monte-carlo-ppi.github.io/.

Related papers

• Efficient Weight-Space Laplace-Gaussian Filtering and Smoothing for Sequential Deep Learning [29.328769628694484]
  Efficiently learning a sequence of related tasks, such as in continual learning, poses a significant challenge for neural nets. We address this challenge with a grounded framework for sequentially learning related tasks based on Bayesian inference.
  arXiv  Detail & Related papers  (2024-10-09T11:54:33Z)
• Model Predictive Control with Gaussian-Process-Supported Dynamical Constraints for Autonomous Vehicles [82.65261980827594]
  We propose a model predictive control approach for autonomous vehicles that exploits learned Gaussian processes for predicting human driving behavior. A multi-mode predictive control approach considers the possible intentions of the human drivers.
  arXiv  Detail & Related papers  (2023-03-08T17:14:57Z)
• Learning Sampling Distributions for Model Predictive Control [36.82905770866734]
  Sampling-based approaches to Model Predictive Control (MPC) have become a cornerstone of contemporary approaches to MPC. We propose to carry out all operations in the latent space, allowing us to take full advantage of the learned distribution. Specifically, we frame the learning problem as bi-level optimization and show how to train the controller with backpropagation-through-time.
  arXiv  Detail & Related papers  (2022-12-05T20:35:36Z)
• GPU-Accelerated Policy Optimization via Batch Automatic Differentiation of Gaussian Processes for Real-World Control [8.720903734757627]
  We develop a policy optimization method by leveraging fast predictive sampling methods to process batches of trajectories in every forward pass. We demonstrate the effectiveness of our approach in training policies on a set of reference-tracking control experiments with a heavy-duty machine.
  arXiv  Detail & Related papers  (2022-02-28T09:31:15Z)
• Robust Value Iteration for Continuous Control Tasks [99.00362538261972]
  When transferring a control policy from simulation to a physical system, the policy needs to be robust to variations in the dynamics to perform well. We present Robust Fitted Value Iteration, which uses dynamic programming to compute the optimal value function on the compact state domain. We show that robust value is more robust compared to deep reinforcement learning algorithm and the non-robust version of the algorithm.
  arXiv  Detail & Related papers  (2021-05-25T19:48:35Z)
• Active and sparse methods in smoothed model checking [2.28438857884398]
  We consider extensions to smoothed model checking based on sparse variational methods and active learning. Online extensions of sparse variational Gaussian process inference algorithms are demonstrated to provide a scalable method for implementing active learning approaches for smoothed model checking.
  arXiv  Detail & Related papers  (2021-04-20T13:03:25Z)
• Gaussian Process-based Min-norm Stabilizing Controller for Control-Affine Systems with Uncertain Input Effects and Dynamics [90.81186513537777]
  We propose a novel compound kernel that captures the control-affine nature of the problem. We show that this resulting optimization problem is convex, and we call it Gaussian Process-based Control Lyapunov Function Second-Order Cone Program (GP-CLF-SOCP)
  arXiv  Detail & Related papers  (2020-11-14T01:27:32Z)
• Global Optimization of Gaussian processes [52.77024349608834]
  We propose a reduced-space formulation with trained Gaussian processes trained on few data points. The approach also leads to significantly smaller and computationally cheaper sub solver for lower bounding. In total, we reduce time convergence by orders of orders of the proposed method.
  arXiv  Detail & Related papers  (2020-05-21T20:59:11Z)
• Localized active learning of Gaussian process state space models [63.97366815968177]
  A globally accurate model is not required to achieve good performance in many common control applications. We propose an active learning strategy for Gaussian process state space models that aims to obtain an accurate model on a bounded subset of the state-action space. By employing model predictive control, the proposed technique integrates information collected during exploration and adaptively improves its exploration strategy.
  arXiv  Detail & Related papers  (2020-05-04T05:35:02Z)
• Active Learning for Gaussian Process Considering Uncertainties with Application to Shape Control of Composite Fuselage [7.358477502214471]
  We propose two new active learning algorithms for the Gaussian process with uncertainties. We show that the proposed approach can incorporate the impact from uncertainties, and realize better prediction performance. This approach has been applied to improving the predictive modeling for automatic shape control of composite fuselage.
  arXiv  Detail & Related papers  (2020-04-23T02:04:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.