MODIP: Efficient Model-Based Optimization for Diffusion Policies

• URL: http://arxiv.org/abs/2606.10825v1
• Date: Tue, 09 Jun 2026 13:09:21 GMT
• Title: MODIP: Efficient Model-Based Optimization for Diffusion Policies
• Abstract summary: Diffusion policies (DPs) have emerged as expressive policy representations for robot learning.<n>We propose MODIP, a framework for the offline-to-online fine-tuning of DPs.
• Score: 13.033386079956266
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Diffusion policies (DPs) have emerged as expressive policy representations for robot learning, often used with imitation learning methods such as behavioral cloning (BC). However, while their success has largely been confined to BC, direct reinforcement learning (RL) fine-tuning remains challenging because actions are generated through a multi-step denoising process. In this work, we propose MODIP, a framework for the offline-to-online fine-tuning of DPs. Rather than directly applying RL to the DPs, MODIP leverages a world model (WM) to guide policy adaptation and keeps the simplicity and stability of BC. We utilize model predictive control (MPC) to generate high-quality trajectories within the WM, and use them as supervised targets for fine-tuning the DP. To make MPC planning efficient, MODIP uses a terminal state value instead of a policy-dependent state-action value, reducing inference time. Additionally, MODIP trains critics with policy-independent TD targets, reducing training time. Experiments on D4RL (MuJoCo, Kitchen) and RoboMimic tasks show that MODIP improves diffusion policies beyond BC, and is competitive with or outperforms diffusion policy RL fine-tuning methods and strong model-based baselines such as TD-MPC2.

Related papers

• The Mirage of Optimizing Training Policies: Monotonic Inference Policies as the Real Objective for LLM Reinforcement Learning [65.72688364382331]
  Reinforcement learning (RL) has gained growing attention in large language model (LLM) post-training, yet RL training remains fragile and can suffer from instability or collapse.<n>One vital cause is training-inference mismatch: LLM adopts separate inference and training engines for generation efficiency and training precision.<n>We introduce Monotonic Inference Policy Update (MIPU), a two-step LLM RL framework that constructs sampler-referenced candidate updates and selectively accepts synchronized candidates.
  arXiv  Detail & Related papers  (2026-06-28T17:40:02Z)
• HOLO-MPPI: Multi-Scenario Motion Planning via Hierarchical Policy Optimization [8.210283573775568]
  We present HOLO-MPPI, a multi-scenario motion planning framework that combines high-level policy learning with low-level optimal control.<n>Offline, we learn a high-level policy that proposes scenario-robust plans in an abstract action space, with a learned world model for online rollout.<n>Online, the policy serves as a data-driven prior generator that parameterizes MPPI's sampling distribution conditioned on the current observation and goal.<n>MPPI then optimize low-level control sequences around this prior in real time to adapt to local disturbances.
  arXiv  Detail & Related papers  (2026-06-15T09:49:14Z)
• V-GRPO: Online Reinforcement Learning for Denoising Generative Models Is Easier than You Think [90.69263509098948]
  We introduce Variational GRPO, a method that integrates ELBO-based surrogates with the Group Relative Policy Optimization algorithm.<n>V-GRPO achieves state-of-the-art performance in text-to-image synthesis, while delivering a $2times$ speedup over MixGRPO and a $3times$ speedup over DiffusionNFT.
  arXiv  Detail & Related papers  (2026-04-25T17:03:21Z)
• Model Predictive Control with Differentiable World Models for Offline Reinforcement Learning [17.293286354647716]
  We introduce an inference time adaptation framework inspired by model predictive control (MPC)<n>We show that exploiting inference-time information to optimize the policy parameters yields consistent gains over strong offline RL baselines.
  arXiv  Detail & Related papers  (2026-03-23T18:05:29Z)
• Bootstrap Off-policy with World Model [59.129118672069644]
  We propose BOOM, a framework that tightly integrates planning and off-policy learning through a bootstrap loop.<n>BOOM achieves state-of-the-art results in both training stability and final performance.
  arXiv  Detail & Related papers  (2025-11-01T06:33:04Z)
• BAPO: Stabilizing Off-Policy Reinforcement Learning for LLMs via Balanced Policy Optimization with Adaptive Clipping [69.74252624161652]
  We propose BAlanced Policy Optimization with Adaptive Clipping (BAPO)<n>BAPO dynamically adjusts clipping bounds to adaptively re-balance positive and negative contributions, preserve entropy, and stabilize RL optimization.<n>On AIME 2024 and AIME 2025 benchmarks, our 7B BAPO model surpasses open-source counterparts such as SkyWork-OR1-7B.
  arXiv  Detail & Related papers  (2025-10-21T12:55:04Z)
• Computationally efficient Gauss-Newton reinforcement learning for model predictive control [0.8437187555622164]
  We introduce a Gauss-Newton approximation of the deterministic policy Hessian that eliminates the need for second-order policy derivatives.<n>We demonstrate the effectiveness of the approach on a nonlinear continuously stirred tank reactor.
  arXiv  Detail & Related papers  (2025-08-04T14:00:40Z)
• Steering Your Diffusion Policy with Latent Space Reinforcement Learning [46.598122553180005]
  Behavioral cloning (BC)-learned policies typically require collecting additional human demonstrations to further improve their behavior.<n> reinforcement learning (RL) holds the promise of enabling autonomous online policy improvement, but often falls short of achieving this due to the large number of samples it typically requires.<n>We show that DSRL is highly sample efficient, requires only black-box access to the BC policy, and enables effective real-world autonomous policy improvement.
  arXiv  Detail & Related papers  (2025-06-18T18:35:57Z)
• Bootstrapped Model Predictive Control [19.652808098339644]
  We introduce Bootstrapped Model Predictive Control (BMPC), a novel algorithm that performs policy learning in a bootstrapped manner.<n>BMPC learns a network policy by imitating an MPC expert, and in turn, uses this policy to guide the MPC process.<n>Our method achieves superior performance over prior works on diverse continuous control tasks.
  arXiv  Detail & Related papers  (2025-03-24T16:46:36Z)
• Policy Agnostic RL: Offline RL and Online RL Fine-Tuning of Any Class and Backbone [72.17534881026995]
  We develop an offline and online fine-tuning approach called policy-agnostic RL (PA-RL)<n>We show the first result that successfully fine-tunes OpenVLA, a 7B generalist robot policy, autonomously with Cal-QL, an online RL fine-tuning algorithm.
  arXiv  Detail & Related papers  (2024-12-09T17:28:03Z)
• Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning [70.20191211010847]
  Offline reinforcement learning (RL) aims to learn an optimal policy using a previously collected static dataset. We introduce Diffusion Q-learning (Diffusion-QL) that utilizes a conditional diffusion model to represent the policy. We show that our method can achieve state-of-the-art performance on the majority of the D4RL benchmark tasks.
  arXiv  Detail & Related papers  (2022-08-12T09:54:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.