Real-World Reinforcement Learning with MPC Scaffolding for Dexterous Manipulation

• URL: http://arxiv.org/abs/2609.14878v1
• Date: Mon, 14 Sep 2026 00:57:55 GMT
• Title: Real-World Reinforcement Learning with MPC Scaffolding for Dexterous Manipulation
• Abstract summary: We propose a framework that uses sampling-based model predictive control (MPC) as scaffolding for real-world dexterous RL.<n>MPC is used to provide structured prior experience and task-directed guidance during learning without human demonstrations or corrective actions.<n>We show that the framework enables efficient, low-orientation, real-world dexterous RL.
• Score: 3.693403461852542
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Real-world reinforcement learning (RL) offers a promising route to dexterous manipulation policies that can adapt directly from physical interaction, but learning is hindered by inefficient early exploration and costly failures. We propose a framework that uses sampling-based model predictive control (MPC) as scaffolding for real-world dexterous RL, providing structured prior experience and task-directed guidance during learning without human demonstrations or corrective actions. A small set of MPC trajectories is first used to populate an offline replay buffer and to pretrain the actor and critic. During online learning, MPC intermittently guides data collection while an off-policy Soft Actor-Critic learner trains from both prior MPC experience and newly collected physical interaction, with control gradually transitioning to the learned policy. On continuous in-hand rotation with a 16-DoF Allegro hand, the method reaches 100\% success in policy-only evaluation (5/5 trials) after 7 minutes of online RL, following initialization with 20 MPC trajectories collected on hardware in 12 minutes. Online training incurs about three object drops on average. After 20 minutes of online learning, the policy achieves more than five times the rotation speed of the MPC controller. It completes 1000 consecutive rotations over more than 110 minutes without a drop. Ablations show complementary benefits from MPC-based pretraining, retained MPC experience, and online MPC guidance. We further demonstrate rapid adaptation to different object geometries and successful goal-conditioned reorientation, showing that the framework enables efficient, low-intervention, real-world dexterous RL.

Related papers

• Accelerating and Scaling MPC-Guided Reinforcement Learning for Humanoid Locomotion and Manipulation [18.311232755848888]
  In humanoid motion control, model predictive control (MPC) offers physically grounded prediction and constraint handling.<n> reinforcement learning (RL) enables robust whole-body skills through large-scale simulation.<n>This work studies efficient training-time MPC guidance for humanoid locomotion and manipulation, termed MPC-RL.
  arXiv  Detail & Related papers  (2026-06-04T04:12:47Z)
• Beyond Conservative Automated Driving in Multi-Agent Scenarios via Coupled Model Predictive Control and Deep Reinforcement Learning [7.804905867413516]
  Model Predictive Control (MPC) offers structured constraint handling through optimization.<n>Deep Reinforcement Learning (RL) learns adaptive behaviors from experience but often struggles with safety assurance.<n>We present an integrated MPC-RL framework to improve navigation performance in multi-agent scenarios.
  arXiv  Detail & Related papers  (2026-04-15T13:58:38Z)
• MePo: Meta Post-Refinement for Rehearsal-Free General Continual Learning [47.195830952416294]
  We introduce an innovative approach named Meta Post-Refinement (MePo) for PTMs-based general continual learning (GCL)<n>MePo constructs pseudo task sequences from pretraining data and develops a bi-level meta-learning paradigm to refine the pretrained backbone.<n>MePo serves as a plug-in strategy that achieves significant performance gains across a variety of GCL benchmarks and pretrained checkpoints.
  arXiv  Detail & Related papers  (2026-02-08T12:15:35Z)
• Fine-Tuning of Neural Network Approximate MPC without Retraining via Bayesian Optimization [81.35990332700389]
  Approximate model-predictive control (AMPC) aims to imitate an MPC's behavior with a neural network.<n>During deployment, the parameters of the underlying MPC must usually be fine-tuned.<n>Recent work addresses this problem by adapting AMPC without retraining using approximated sensitivities of the MPC's optimization problem.
  arXiv  Detail & Related papers  (2025-12-16T12:24:08Z)
• Human-in-the-loop Online Rejection Sampling for Robotic Manipulation [55.99788088622936]
  Hi-ORS stabilizes value estimation by filtering out negatively rewarded samples during online fine-tuning.<n>Hi-ORS fine-tunes a pi-base policy to master contact-rich manipulation in just 1.5 hours of real-world training.
  arXiv  Detail & Related papers  (2025-10-30T11:53:08Z)
• UI-S1: Advancing GUI Automation via Semi-online Reinforcement Learning [78.86567400365392]
  We present Semi-online Reinforcement Learning, a novel paradigm that simulates online RL on offline trajectories.<n>To capture long-term training signals, Semi-online RL introduces discounted future returns into the reward computation.<n>Experiments show that ours Semi-online RL achieves SOTA performance among 7B models across four dynamic benchmarks.
  arXiv  Detail & Related papers  (2025-09-15T03:24:08Z)
• Fast Adaptation with Behavioral Foundation Models [82.34700481726951]
  Unsupervised zero-shot reinforcement learning has emerged as a powerful paradigm for pretraining behavioral foundation models.<n>Despite promising results, zero-shot policies are often suboptimal due to errors induced by the unsupervised training process.<n>We propose fast adaptation strategies that search in the low-dimensional task-embedding space of the pre-trained BFM to rapidly improve the performance of its zero-shot policies.
  arXiv  Detail & Related papers  (2025-04-10T16:14:17Z)
• Toward Scalable Multirobot Control: Fast Policy Learning in Distributed MPC [22.644778818620185]
  This article proposes a novel distributed learning-based predictive control (DLPC) framework for scalable multirobot control.<n>Unlike conventional DMPC methods that calculate open-loop control sequences, our approach generates explicit closed-loop DMPC policies for MRS without using numerical solvers.<n>The learned control policies could be deployed online to MRS with varying robot scales, enhancing scalability and transferability for large-scale MRS.
  arXiv  Detail & Related papers  (2024-12-27T14:31:52Z)
• FastRLAP: A System for Learning High-Speed Driving via Deep RL and Autonomous Practicing [71.76084256567599]
  We present a system that enables an autonomous small-scale RC car to drive aggressively from visual observations using reinforcement learning (RL) Our system, FastRLAP (faster lap), trains autonomously in the real world, without human interventions, and without requiring any simulation or expert demonstrations. The resulting policies exhibit emergent aggressive driving skills, such as timing braking and acceleration around turns and avoiding areas which impede the robot's motion, approaching the performance of a human driver using a similar first-person interface over the course of training.
  arXiv  Detail & Related papers  (2023-04-19T17:33:47Z)
• Model Predictive Control via On-Policy Imitation Learning [28.96122879515294]
  We develop new sample complexity results and performance guarantees for data-driven Model Predictive Control. Our algorithm uses the structure of constrained linear MPC, and our analysis uses the properties of the explicit MPC solution to theoretically bound the number of online MPC trajectories needed to achieve optimal performance.
  arXiv  Detail & Related papers  (2022-10-17T16:06:06Z)
• AWAC: Accelerating Online Reinforcement Learning with Offline Datasets [84.94748183816547]
  We show that our method, advantage weighted actor critic (AWAC), enables rapid learning of skills with a combination of prior demonstration data and online experience. Our results show that incorporating prior data can reduce the time required to learn a range of robotic skills to practical time-scales.
  arXiv  Detail & Related papers  (2020-06-16T17:54:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.