Dream-MPC: Gradient-Based Model Predictive Control with Latent Imagination

• URL: http://arxiv.org/abs/2605.04568v1
• Date: Wed, 06 May 2026 07:13:11 GMT
• Title: Dream-MPC: Gradient-Based Model Predictive Control with Latent Imagination
• Abstract summary: State-of-the-art model-based Reinforcement Learning (RL) approaches either use gradient-free, population-based methods for planning, learned policy networks, or a combination of policy networks and planning.<n>We propose Dream-MPC, a novel approach that generates few candidate trajectories from a rolled-out policy and optimize each trajectory by gradient ascent using a learned world model.<n>Our results on 24 continuous control tasks show that Dream-MPC can significantly improve the performance of the underlying policy and can outperform gradient-free MPC and state-of-the-art baselines.
• Score: 17.4778210092639
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: State-of-the-art model-based Reinforcement Learning (RL) approaches either use gradient-free, population-based methods for planning, learned policy networks, or a combination of policy networks and planning. Hybrid approaches that combine Model Predictive Control (MPC) with a learned model and a policy prior to leverage the advantages of both paradigms have shown promising results. However, these approaches typically rely on gradient-free optimization methods, which can be computationally expensive for high-dimensional control tasks. While gradient-based methods are a promising alternative, recent works have empirically shown that gradient-based methods often perform worse than their gradient-free counterparts. We propose Dream-MPC, a novel approach that generates few candidate trajectories from a rolled-out policy and optimizes each trajectory by gradient ascent using a learned world model, uncertainty regularization and amortization of optimization iterations over time by reusing previously optimized actions. Our results on 24 continuous control tasks show that Dream-MPC can significantly improve the performance of the underlying policy and can outperform gradient-free MPC and state-of-the-art baselines. We will open source our code and more at https://dream-mpc.github.io.

Related papers

• A Continuous-Time Reinforcement Learning Framework for Fine-Tuning Discrete Diffusion Models [9.133661763917374]
  We formulate reinforcement learning (RL) in continuous time with discrete state spaces and possibly arbitrary action spaces.<n>We develop a complete continuous-time RL framework for fine-tuning score-based discrete diffusion models.
  arXiv  Detail & Related papers  (2026-07-16T03:19:53Z)
• V-GRPO: Online Reinforcement Learning for Denoising Generative Models Is Easier than You Think [90.69263509098948]
  We introduce Variational GRPO, a method that integrates ELBO-based surrogates with the Group Relative Policy Optimization algorithm.<n>V-GRPO achieves state-of-the-art performance in text-to-image synthesis, while delivering a $2times$ speedup over MixGRPO and a $3times$ speedup over DiffusionNFT.
  arXiv  Detail & Related papers  (2026-04-25T17:03:21Z)
• Closing the Train-Test Gap in World Models for Gradient-Based Planning [64.36544881136405]
  We propose improved methods for training world models that enable efficient gradient-based planning.<n>At test time, our approach outperforms or matches the classical gradient-free cross-entropy method.
  arXiv  Detail & Related papers  (2025-12-10T18:59:45Z)
• Stabilizing Policy Gradients for Sample-Efficient Reinforcement Learning in LLM Reasoning [77.92320830700797]
  Reinforcement Learning has played a central role in enabling reasoning capabilities of Large Language Models.<n>We propose a tractable computational framework that tracks and leverages curvature information during policy updates.<n>The algorithm, Curvature-Aware Policy Optimization (CAPO), identifies samples that contribute to unstable updates and masks them out.
  arXiv  Detail & Related papers  (2025-10-01T12:29:32Z)
• Unifying Model Predictive Path Integral Control, Reinforcement Learning, and Diffusion Models for Optimal Control and Planning [6.871390204787483]
  We establish a unified perspective that connects MPPI, RL, and Diffusion Models through gradient-based optimization on the Gibbs measure.<n>We first show that MPPI can be interpreted as performing gradient ascent on a smoothed energy function.<n>We then demonstrate that Policy Gradient methods reduce to MPPI by applying an exponential transformation to the objective function.
  arXiv  Detail & Related papers  (2025-02-27T19:26:36Z)
• Gradient-based Planning with World Models [21.9392160209565]
  We present an exploration of a gradient-based alternative that fully leverages the differentiability of the world model. In a sample-efficient setting, our method achieves on par or superior performance compared to the alternative approaches in most tasks.
  arXiv  Detail & Related papers  (2023-12-28T18:54:21Z)
• Acceleration in Policy Optimization [50.323182853069184]
  We work towards a unifying paradigm for accelerating policy optimization methods in reinforcement learning (RL) by integrating foresight in the policy improvement step via optimistic and adaptive updates. We define optimism as predictive modelling of the future behavior of a policy, and adaptivity as taking immediate and anticipatory corrective actions to mitigate errors from overshooting predictions or delayed responses to change. We design an optimistic policy gradient algorithm, adaptive via meta-gradient learning, and empirically highlight several design choices pertaining to acceleration, in an illustrative task.
  arXiv  Detail & Related papers  (2023-06-18T15:50:57Z)
• Multi-Objective Policy Gradients with Topological Constraints [108.10241442630289]
  We present a new algorithm for a policy gradient in TMDPs by a simple extension of the proximal policy optimization (PPO) algorithm. We demonstrate this on a real-world multiple-objective navigation problem with an arbitrary ordering of objectives both in simulation and on a real robot.
  arXiv  Detail & Related papers  (2022-09-15T07:22:58Z)
• Anchor-Changing Regularized Natural Policy Gradient for Multi-Objective Reinforcement Learning [17.916366827429034]
  We study policy optimization for Markov decision processes (MDPs) with multiple reward value functions. We propose an Anchor-changing Regularized Natural Policy Gradient framework, which can incorporate ideas from well-performing first-order methods.
  arXiv  Detail & Related papers  (2022-06-10T21:09:44Z)
• Evaluating model-based planning and planner amortization for continuous control [79.49319308600228]
  We take a hybrid approach, combining model predictive control (MPC) with a learned model and model-free policy learning. We find that well-tuned model-free agents are strong baselines even for high DoF control problems. We show that it is possible to distil a model-based planner into a policy that amortizes the planning without any loss of performance.
  arXiv  Detail & Related papers  (2021-10-07T12:00:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.