Set-Supervised Diffusion Policy: Learning Action-Chunking Diffusion through Corrections

• URL: http://arxiv.org/abs/2606.01865v1
• Date: Mon, 01 Jun 2026 08:14:38 GMT
• Title: Set-Supervised Diffusion Policy: Learning Action-Chunking Diffusion through Corrections
• Abstract summary: Diffusion policies provide paired supervision in the form of the robot's undesired actions and the human teacher's corrective actions.<n>We propose Set-Supervised Diffusion Policy (SDP), a novel learning framework that utilizes contrastive action-chunk data to train diffusion policies from human corrections.<n>SDP consistently improves policy performance, with particularly strong gains in robustness to noisy data.
• Score: 25.961145214027255
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Diffusion policies have recently emerged as a powerful framework for robotic manipulation. However, like other behavior cloning methods, they remain vulnerable to distributional shift, often requiring human-in-the-loop interventions to correct failures during deployment. These interactions naturally provide paired supervision in the form of the robot's undesired actions and the human teacher's corrective actions. Yet existing data aggregation pipelines and standard behavior cloning losses largely ignore this negative signal from undesired actions, leading to overfitting to teacher's actions and an increasing reliance on costly expert data. To address this limitation, we propose Set-Supervised Diffusion Policy (SDP), a novel learning framework that utilizes contrastive action-chunk data to train diffusion policies from human corrections. From paired positive and negative action-chunks, SDP constructs a set of desired action-chunks and designs a training pipeline that encourages the diffusion policy to align with the set. Through extensive experiments across multiple robotic manipulation tasks, we demonstrate that SDP consistently improves policy performance, with particularly strong gains in robustness to noisy data. Moreover, SDP induces high-quality aggregated datasets, enabling more efficient and reliable policy learning from human-in-the-loop corrections. Our code is available at https://set-supervised-diffusion-policy.github.io/.

Related papers

• PAC-DP: PAC-Bayesian Diffusion Policy Learning [13.10166905442267]
  Diffusion Policies (DPs) are able to perform complex manipulation tasks.<n>We propose PAC-DP, an approach that increases the performance of DPs in robotic manipulation tasks.
  arXiv  Detail & Related papers  (2026-07-27T11:43:49Z)
• Value-Guidance MeanFlow for Offline Multi-Agent Reinforcement Learning [42.476656442284835]
  Offline multi-agent reinforcement learning (MARL) aims to learn the optimal joint policy from pre-collected datasets.<n>We propose Value Guidance Multi-agent MeanFlow Policy (VGM$2$P), a simple yet effective flow-based policy learning framework.<n>VGM$2$P uses global advantage values to guide agent collaboration, treating optimal policy learning as conditional behavior cloning.
  arXiv  Detail & Related papers  (2026-04-09T12:31:43Z)
• REFINE-DP: Diffusion Policy Fine-tuning for Humanoid Loco-manipulation via Reinforcement Learning [23.094280494500214]
  We present REFINE-DP (REinforcement learning FINE-tuning of Diffusion Policy), a hierarchical framework that jointly optimize a DP high-level planner and an RL-based low-level loco-manipulation controller.<n> REFINE-DP achieves an over $90%$ success rate in simulation, even in out-of-distribution cases not seen in the pre-trained data.
  arXiv  Detail & Related papers  (2026-03-14T02:21:19Z)
• PPGuide: Steering Diffusion Policies with Performance Predictive Guidance [28.698103923760172]
  PPGuide steers a pre-trained diffusion policy away from failure modes at inference time.<n>It uses attention-based multiple instance learning to automatically estimate which observation-action chunks from the policy's rollouts are relevant to success or failure.<n>We validated our proposed PPGuide across a diverse set of tasks from the Robomimic and MimicGen benchmarks.
  arXiv  Detail & Related papers  (2026-03-11T17:10:16Z)
• RFS: Reinforcement Learning with Residual Flow Steering for Dexterous Manipulation [7.500999283386335]
  Residual Flow Steering (RFS) is a data-efficient reinforcement learning framework for adapting pretrained generative policies.<n>RFS steers a pretrained flow-matching policy by jointly optimizing a residual action and a latent noise distribution.<n>We demonstrate the effectiveness of RFS on dexterous manipulation tasks, showing efficient fine-tuning in both simulation and real-world settings.
  arXiv  Detail & Related papers  (2026-02-02T08:11:57Z)
• Steering Vision-Language-Action Models as Anti-Exploration: A Test-Time Scaling Approach [78.4812458793128]
  We propose textbfTACO, a test-time-scaling framework that applies a lightweight pseudo-count estimator as a high-fidelity verifier of action chunks.<n>Our method resembles the classical anti-exploration principle in offline reinforcement learning (RL), and being gradient-free, it incurs significant computational benefits.
  arXiv  Detail & Related papers  (2025-12-02T14:42:54Z)
• FDPP: Fine-tune Diffusion Policy with Human Preference [57.44575105114056]
  Fine-tuning Diffusion Policy with Human Preference learns a reward function through preference-based learning.<n>This reward is then used to fine-tune the pre-trained policy with reinforcement learning.<n>Experiments demonstrate that FDPP effectively customizes policy behavior without compromising performance.
  arXiv  Detail & Related papers  (2025-01-14T17:15:27Z)
• Inference-Time Policy Steering through Human Interactions [54.02655062969934]
  During inference, humans are often removed from the policy execution loop.<n>We propose an Inference-Time Policy Steering framework that leverages human interactions to bias the generative sampling process.<n>Our proposed sampling strategy achieves the best trade-off between alignment and distribution shift.
  arXiv  Detail & Related papers  (2024-11-25T18:03:50Z)
• Let Offline RL Flow: Training Conservative Agents in the Latent Space of Normalizing Flows [58.762959061522736]
  offline reinforcement learning aims to train a policy on a pre-recorded and fixed dataset without any additional environment interactions. We build upon recent works on learning policies in latent action spaces and use a special form of Normalizing Flows for constructing a generative model. We evaluate our method on various locomotion and navigation tasks, demonstrating that our approach outperforms recently proposed algorithms.
  arXiv  Detail & Related papers  (2022-11-20T21:57:10Z)
• Latent-Variable Advantage-Weighted Policy Optimization for Offline RL [70.01851346635637]
  offline reinforcement learning methods hold the promise of learning policies from pre-collected datasets without the need to query the environment for new transitions. In practice, offline datasets are often heterogeneous, i.e., collected in a variety of scenarios. We propose to leverage latent-variable policies that can represent a broader class of policy distributions. Our method improves the average performance of the next best-performing offline reinforcement learning methods by 49% on heterogeneous datasets.
  arXiv  Detail & Related papers  (2022-03-16T21:17:03Z)
• Human-in-the-Loop Imitation Learning using Remote Teleoperation [72.2847988686463]
  We build a data collection system tailored to 6-DoF manipulation settings. We develop an algorithm to train the policy iteratively on new data collected by the system. We demonstrate that agents trained on data collected by our intervention-based system and algorithm outperform agents trained on an equivalent number of samples collected by non-interventional demonstrators.
  arXiv  Detail & Related papers  (2020-12-12T05:30:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.