A Single Diffusion-Policy Controller for Multi-Task Block Pushing with Zero-Shot Sim-to-Real Transfer

• URL: http://arxiv.org/abs/2607.10892v1
• Date: Sun, 12 Jul 2026 19:48:46 GMT
• Title: A Single Diffusion-Policy Controller for Multi-Task Block Pushing with Zero-Shot Sim-to-Real Transfer
• Abstract summary: We explore training diffusion policies from scratch using reinforcement learning (RL) for multi-task robotic manipulation.<n>We aim to train a single diffusion policy for block-pushing tasks with multiple shapes.<n>We evaluate whether the trained diffusion policy transfers in zero-shot to real-world tasks under varying environmental conditions.
• Score: 15.906272895307664
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Diffusion policies have shown promising empirical performance in representing and learning complex maneuvers for robots using behavior cloning (BC). In this paper, we explore training diffusion policies from scratch using reinforcement learning (RL) for multi-task robotic manipulation. Specifically, we aim to train a single diffusion policy for block-pushing tasks with multiple shapes. The proposed framework features a simple policy loss function, which is a reweighted evidence lower bound used in BC-based diffusion policy training and can seamlessly serve as the policy learning module in RL algorithms. To address the exploration challenges arising from the absence of demonstrations, we incorporate reverse curriculum generation and objective-centric representations. Combined with the expressiveness of diffusion policies, our design supports learning of multi-task block-pushing policies in our sparse-reward simulation setting. We further evaluate whether the trained diffusion policy transfers in zero-shot to real-world tasks under varying environmental conditions including goal positions, block shapes, block weights and surface friction, providing evidence that this pipeline can transfer to our real-world block-pushing setup under the tested variations.

Related papers

• Diffusion Policy through Conditional Proximal Policy Optimization [6.836651088754774]
  diffusion policies have shown strong potential in modeling multi-modal behaviors.<n>A key challenge is the difficulty of computing action log-likelihood under the diffusion model.<n>We propose a novel and efficient method to train a diffusion policy in an on-policy setting.
  arXiv  Detail & Related papers  (2026-03-05T04:12:13Z)
• Decentralized Learning Strategies for Estimation Error Minimization with Graph Neural Networks [86.99017195607077]
  We address real-time sampling and estimation of autoregressive Markovian sources in wireless networks.<n>We propose a graphical reinforcement learning framework for policy optimization.<n>Theoretically, our proposed policies are transferable, allowing a policy trained on one graph to be effectively applied to structurally similar graphs.
  arXiv  Detail & Related papers  (2026-01-19T02:18:45Z)
• Zero-Shot Policy Transfer in Reinforcement Learning using Buckingham's Pi Theorem [42.37643072381109]
  Reinforcement learning policies often fail to generalize to new robots, tasks, or environments with different physical parameters.<n>This paper presents a simple, zero-shot transfer method based on Buckingham's Pi Theorem to address this limitation.
  arXiv  Detail & Related papers  (2025-10-09T19:36:18Z)
• Reinforcement Learning with Discrete Diffusion Policies for Combinatorial Action Spaces [57.466101098183884]
  Reinforcement learning (RL) struggles to scale to large, action spaces common in many real-world problems.<n>This paper introduces a novel framework for training discrete diffusion models as highly effective policies in complex settings.
  arXiv  Detail & Related papers  (2025-09-26T21:53:36Z)
• Robot Policy Transfer with Online Demonstrations: An Active Reinforcement Learning Approach [3.637365301757111]
  We present Policy Transfer with Online Demonstrations, an active Learning from Demonstration algorithm for policy transfer.<n>We evaluate our method in eight robotic scenarios, involving policy transfer across diverse environment characteristics.<n>Results show that our method significantly outperforms all baselines in terms of average success rate and sample efficiency.
  arXiv  Detail & Related papers  (2025-03-17T09:47:42Z)
• From Imitation to Refinement -- Residual RL for Precise Assembly [19.9786629249219]
  Recent advances in Behavior Cloning (BC) have made it easy to teach robots new tasks.<n>However, we find that the ease of teaching comes at the cost of unreliable performance.<n>We devise a simple yet effective method, ResiP, that overcomes the reliability problem while retaining BC's ease of teaching and long-horizon capabilities.
  arXiv  Detail & Related papers  (2024-07-23T17:44:54Z)
• Learning Multimodal Behaviors from Scratch with Diffusion Policy Gradient [26.675822002049372]
  Deep Diffusion Policy Gradient (DDiffPG) is a novel actor-critic algorithm that learns from scratch multimodal policies. DDiffPG forms a multimodal training batch and utilizes mode-specific Q-learning to mitigate the inherent greediness of the RL objective. Our approach further allows the policy to be conditioned on mode-specific embeddings to explicitly control the learned modes.
  arXiv  Detail & Related papers  (2024-06-02T09:32:28Z)
• Decentralized Learning Strategies for Estimation Error Minimization with Graph Neural Networks [86.99017195607077]
  We address the challenge of sampling and remote estimation for autoregressive Markovian processes in a wireless network with statistically-identical agents.<n>Our goal is to minimize time-average estimation error and/or age of information with decentralized scalable sampling and transmission policies.
  arXiv  Detail & Related papers  (2024-04-04T06:24:11Z)
• Distilling Reinforcement Learning Policies for Interpretable Robot Locomotion: Gradient Boosting Machines and Symbolic Regression [53.33734159983431]
  This paper introduces a novel approach to distill neural RL policies into more interpretable forms. We train expert neural network policies using RL and distill them into (i) GBMs, (ii) EBMs, and (iii) symbolic policies.
  arXiv  Detail & Related papers  (2024-03-21T11:54:45Z)
• Co-learning Planning and Control Policies Constrained by Differentiable Logic Specifications [4.12484724941528]
  This paper presents a novel reinforcement learning approach to solving high-dimensional robot navigation tasks. We train high-quality policies with much fewer samples compared to existing reinforcement learning algorithms. Our approach also demonstrates capabilities for high-dimensional control and avoiding suboptimal policies via policy alignment.
  arXiv  Detail & Related papers  (2023-03-02T15:24:24Z)
• Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning [70.20191211010847]
  Offline reinforcement learning (RL) aims to learn an optimal policy using a previously collected static dataset. We introduce Diffusion Q-learning (Diffusion-QL) that utilizes a conditional diffusion model to represent the policy. We show that our method can achieve state-of-the-art performance on the majority of the D4RL benchmark tasks.
  arXiv  Detail & Related papers  (2022-08-12T09:54:11Z)
• Composable Learning with Sparse Kernel Representations [110.19179439773578]
  We present a reinforcement learning algorithm for learning sparse non-parametric controllers in a Reproducing Kernel Hilbert Space. We improve the sample complexity of this approach by imposing a structure of the state-action function through a normalized advantage function. We demonstrate the performance of this algorithm on learning obstacle-avoidance policies in multiple simulations of a robot equipped with a laser scanner while navigating in a 2D environment.
  arXiv  Detail & Related papers  (2021-03-26T13:58:23Z)
• A Decentralized Policy Gradient Approach to Multi-task Reinforcement Learning [13.733491423871383]
  We develop a framework for solving multi-task reinforcement learning problems. The goal is to learn a common policy that operates effectively in different environments. We highlight two fundamental challenges in MTRL that are not present in its single task counterpart.
  arXiv  Detail & Related papers  (2020-06-08T03:28:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.