Trust-Region Diffusion Policies for Massively Parallel On-Policy RL

• URL: http://arxiv.org/abs/2606.15260v1
• Date: Sat, 13 Jun 2026 11:35:26 GMT
• Title: Trust-Region Diffusion Policies for Massively Parallel On-Policy RL
• Abstract summary: Diffusion models provide a more expressive policy class and have shown strong performance on challenging control problems.<n>Most diffusion-based RL methods are designed for offline or off-policy training.<n>Trust-region Diffusion Policies (TruDi) enables diffusion policies for on-policy RL with massively parallel simulations.
• Score: 24.462107727661223
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Reinforcement learning with massively parallel simulations has become a standard framework for developing robust, deployable policies; however, most existing approaches still rely on simple Gaussian policy parameterizations. Diffusion models provide a more expressive policy class and have shown strong performance on challenging control problems, yet most diffusion-based RL methods are designed for offline or off-policy training. In this work, we ask whether diffusion policies can be trained effectively in the massively parallel, on-policy regime. To this end, we introduce Trust-region Diffusion Policies (TruDi), which enables diffusion policies for on-policy RL with massively parallel simulations. This setting is particularly challenging because the data distribution changes quickly across updates, making stable training with complex policies difficult. TruDi addresses this by integrating a trust-region optimization rule to enforce a KL-divergence constraint over the entire diffusion trajectory. Empirically, we evaluate TruDi on a diverse set of 4 massively parallel RL benchmarks comprising a total of 73 tasks. Across these tasks, TruDi consistently outperforms or is on-par with strong baselines on standard tasks and achieves clear gains on more challenging humanoid control tasks, establishing a strong new baseline for massively parallel on-policy RL.

Related papers

• Scalable Maximum Entropy Reinforcement Learning for Diffusion Policies via Adjoint Matching [25.3394501858201]
  Diffusion policies represent complex action distributions in reinforcement learning (RL)<n>We introduce a highly efficient algorithm for optimizing diffusion policies by leveraging recent advances in optimal control.<n>Our approach is based on adjoint matching, which enables simulation-free training and circumvents the need for explicit likelihood estimation.
  arXiv  Detail & Related papers  (2026-06-21T18:32:25Z)
• Aligning Flow Map Policies with Optimal Q-Guidance [50.514994916864275]
  Flow map policies are designed for fast action generation by learning to take arbitrary-size jumps.<n>We derive FLOW MAP Q-GUIDANCE (FMQ), a principled closed-form learning target that is optimal for adapting offline flow map policies under a critic-guided trust-region constraint.<n>FMQ achieves state-of-the-art performance in offline-to-online RL, outperforming the previous one-step policy MVP by a relative improvement of 21.3% on the average success rate.
  arXiv  Detail & Related papers  (2026-05-12T17:12:29Z)
• Flow-based Policy With Distributional Reinforcement Learning in Trajectory Optimization [8.371088557371236]
  Reinforcement Learning (RL) has proven highly effective in addressing complex control and decision-making tasks.<n>We propose a RL algorithm termed flow-based policy with distributional RL (FP-DRL)<n>This algorithm models the policy using flow matching, which offers both computational efficiency and the capacity to fit complex distributions.<n>It employs a distributional RL approach to model and optimize the entire return distribution, thereby more effectively guiding multimodal policy updates and improving agent performance.
  arXiv  Detail & Related papers  (2026-04-01T14:47:41Z)
• Dichotomous Diffusion Policy Optimization [46.51375996317989]
  DIPOLE is a novel RL algorithm designed for stable and controllable diffusion policy optimization.<n>We also use DIPOLE to train a large vision-language-action model for end-to-end autonomous driving.
  arXiv  Detail & Related papers  (2025-12-31T16:56:56Z)
• GoRL: An Algorithm-Agnostic Framework for Online Reinforcement Learning with Generative Policies [16.859964356466676]
  GoRL is a framework that optimize a tractable latent policy while utilizing a conditional generative decoder to synthesize actions.<n>GoRL consistently outperforms both Gaussian policies and recent generative-policy baselines.
  arXiv  Detail & Related papers  (2025-12-02T09:49:26Z)
• Reinforcement Learning with Discrete Diffusion Policies for Combinatorial Action Spaces [57.466101098183884]
  Reinforcement learning (RL) struggles to scale to large, action spaces common in many real-world problems.<n>This paper introduces a novel framework for training discrete diffusion models as highly effective policies in complex settings.
  arXiv  Detail & Related papers  (2025-09-26T21:53:36Z)
• Diffusion-based Reinforcement Learning via Q-weighted Variational Policy Optimization [55.97310586039358]
  Diffusion models have garnered widespread attention in Reinforcement Learning (RL) for their powerful expressiveness and multimodality.<n>We propose a novel model-free diffusion-based online RL algorithm, Q-weighted Variational Policy Optimization (QVPO)<n>Specifically, we introduce the Q-weighted variational loss, which can be proved to be a tight lower bound of the policy objective in online RL under certain conditions.<n>We also develop an efficient behavior policy to enhance sample efficiency by reducing the variance of the diffusion policy during online interactions.
  arXiv  Detail & Related papers  (2024-05-25T10:45:46Z)
• Policy Representation via Diffusion Probability Model for Reinforcement Learning [67.56363353547775]
  We build a theoretical foundation of policy representation via the diffusion probability model. We present a convergence guarantee for diffusion policy, which provides a theory to understand the multimodality of diffusion policy. We propose the DIPO which is an implementation for model-free online RL with DIffusion POlicy.
  arXiv  Detail & Related papers  (2023-05-22T15:23:41Z)
• Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning [70.20191211010847]
  Offline reinforcement learning (RL) aims to learn an optimal policy using a previously collected static dataset. We introduce Diffusion Q-learning (Diffusion-QL) that utilizes a conditional diffusion model to represent the policy. We show that our method can achieve state-of-the-art performance on the majority of the D4RL benchmark tasks.
  arXiv  Detail & Related papers  (2022-08-12T09:54:11Z)
• Deep Reinforcement Learning with Robust and Smooth Policy [90.78795857181727]
  We propose to learn a smooth policy that behaves smoothly with respect to states. We develop a new framework -- textbfSmooth textbfRegularized textbfReinforcement textbfLearning ($textbfSR2textbfL$), where the policy is trained with smoothness-inducing regularization. Such regularization effectively constrains the search space, and enforces smoothness in the learned policy.
  arXiv  Detail & Related papers  (2020-03-21T00:10:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.