Accelerating and Scaling MPC-Guided Reinforcement Learning for Humanoid Locomotion and Manipulation

• URL: http://arxiv.org/abs/2606.05687v1
• Date: Thu, 04 Jun 2026 04:12:47 GMT
• Title: Accelerating and Scaling MPC-Guided Reinforcement Learning for Humanoid Locomotion and Manipulation
• Abstract summary: In humanoid motion control, model predictive control (MPC) offers physically grounded prediction and constraint handling.<n> reinforcement learning (RL) enables robust whole-body skills through large-scale simulation.<n>This work studies efficient training-time MPC guidance for humanoid locomotion and manipulation, termed MPC-RL.
• Score: 18.311232755848888
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In humanoid motion control, model predictive control (MPC) offers physically grounded prediction and constraint handling, while reinforcement learning (RL) enables robust whole-body skills through large-scale simulation. However, using MPC inside RL often requires time-consuming problem construction or excessive training overhead, making such frameworks difficult to justify in practice. This work studies efficient training-time MPC guidance for humanoid locomotion and manipulation, termed MPC-RL. We introduce a centroidal-dynamics MPC reward formulation that leverages guidance from MPC trajectories in training time. To make this practical in massively parallel RL, we develop $π^n$MPC, a parallel-in-horizon and construction-free batched GPU MPC solver that operates directly on time-varying dynamics to avoid high memory usage and pre-compilation. Through a variety of comparative studies and hardware validations, we have found that MPC-RL achieves superior performance in locomotion and manipulation skills. The code base is available at https://github.com/junhengl/mpc-rl.

Related papers

• Real-World Reinforcement Learning with MPC Scaffolding for Dexterous Manipulation [3.693403461852542]
  We propose a framework that uses sampling-based model predictive control (MPC) as scaffolding for real-world dexterous RL.<n>MPC is used to provide structured prior experience and task-directed guidance during learning without human demonstrations or corrective actions.<n>We show that the framework enables efficient, low-orientation, real-world dexterous RL.
  arXiv  Detail & Related papers  (2026-09-14T00:57:55Z)
• SRL-MPC: Shape-Aware Reinforcement Learned Model Predictive Control [64.66285242901918]
  Shape-Aware Reinforcement Learned Model Predictive Control (SRL-MPC) is a method for safe, efficient, and adaptive navigation in crowds with heterogeneous shapes without geometry simplification.<n>A reinforcement learning framework then learns a neural policy that reads geometric separation features (GSFs) and outputs real-time MPC parameter updates.<n> Experiments in randomized crowd scenarios with arbitrary shaped robot fleets demonstrate the effectiveness, scalability, and robustness of SRL-MPC.
  arXiv  Detail & Related papers  (2026-08-21T14:46:06Z)
• Cost-Matching Model Predictive Control for Efficient Reinforcement Learning in Humanoid Locomotion [15.125805248577278]
  We propose a cost-matching approach for optimal humanoid locomotion within a Model Predictive Control (MPC)-based Reinforcement Learning framework.<n>A parameterized MPC formulation with centroidal dynamics is trained to approximate the action-value function obtained from high-fidelity closed-loop data.<n>Results demonstrate improved locomotion performance and robustness to model mismatch and external disturbances.
  arXiv  Detail & Related papers  (2026-03-30T10:05:15Z)
• Fine-Tuning of Neural Network Approximate MPC without Retraining via Bayesian Optimization [81.35990332700389]
  Approximate model-predictive control (AMPC) aims to imitate an MPC's behavior with a neural network.<n>During deployment, the parameters of the underlying MPC must usually be fine-tuned.<n>Recent work addresses this problem by adapting AMPC without retraining using approximated sensitivities of the MPC's optimization problem.
  arXiv  Detail & Related papers  (2025-12-16T12:24:08Z)
• MPCritic: A plug-and-play MPC architecture for reinforcement learning [2.217971863441725]
  This paper presents MPCritic, a machine learning-friendly architecture that interfaces seamlessly with MPC tools.<n>MPCritic utilizes the loss landscape defined by a parameterized MPC problem, focusing on "soft" optimization over batched training steps.
  arXiv  Detail & Related papers  (2025-04-01T18:07:07Z)
• Learning to Fly in Seconds [7.259696592534715]
  We show how curriculum learning and a highly optimized simulator enhance sample complexity and lead to fast training times. Our framework enables Simulation-to-Reality (Sim2Real) transfer for direct control after only 18 seconds of training on a consumer-grade laptop.
  arXiv  Detail & Related papers  (2023-11-22T01:06:45Z)
• Model Predictive Control via On-Policy Imitation Learning [28.96122879515294]
  We develop new sample complexity results and performance guarantees for data-driven Model Predictive Control. Our algorithm uses the structure of constrained linear MPC, and our analysis uses the properties of the explicit MPC solution to theoretically bound the number of online MPC trajectories needed to achieve optimal performance.
  arXiv  Detail & Related papers  (2022-10-17T16:06:06Z)
• Real-time Neural-MPC: Deep Learning Model Predictive Control for Quadrotors and Agile Robotic Platforms [59.03426963238452]
  We present Real-time Neural MPC, a framework to efficiently integrate large, complex neural network architectures as dynamics models within a model-predictive control pipeline. We show the feasibility of our framework on real-world problems by reducing the positional tracking error by up to 82% when compared to state-of-the-art MPC approaches without neural network dynamics.
  arXiv  Detail & Related papers  (2022-03-15T09:38:15Z)
• SurRoL: An Open-source Reinforcement Learning Centered and dVRK Compatible Platform for Surgical Robot Learning [78.76052604441519]
  SurRoL is an RL-centered simulation platform for surgical robot learning compatible with the da Vinci Research Kit (dVRK) Ten learning-based surgical tasks are built in the platform, which are common in the real autonomous surgical execution. We evaluate SurRoL using RL algorithms in simulation, provide in-depth analysis, deploy the trained policies on the real dVRK, and show that our SurRoL achieves better transferability in the real world.
  arXiv  Detail & Related papers  (2021-08-30T07:43:47Z)
• AWAC: Accelerating Online Reinforcement Learning with Offline Datasets [84.94748183816547]
  We show that our method, advantage weighted actor critic (AWAC), enables rapid learning of skills with a combination of prior demonstration data and online experience. Our results show that incorporating prior data can reduce the time required to learn a range of robotic skills to practical time-scales.
  arXiv  Detail & Related papers  (2020-06-16T17:54:41Z)
• Information Theoretic Model Predictive Q-Learning [64.74041985237105]
  We present a novel theoretical connection between information theoretic MPC and entropy regularized RL. We develop a Q-learning algorithm that can leverage biased models.
  arXiv  Detail & Related papers  (2019-12-31T00:29:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.