REFINE-DP: Diffusion Policy Fine-tuning for Humanoid Loco-manipulation via Reinforcement Learning

• URL: http://arxiv.org/abs/2603.13707v2
• Date: Tue, 17 Mar 2026 23:57:56 GMT
• Title: REFINE-DP: Diffusion Policy Fine-tuning for Humanoid Loco-manipulation via Reinforcement Learning
• Abstract summary: We present REFINE-DP (REinforcement learning FINE-tuning of Diffusion Policy), a hierarchical framework that jointly optimize a DP high-level planner and an RL-based low-level loco-manipulation controller.<n> REFINE-DP achieves an over $90%$ success rate in simulation, even in out-of-distribution cases not seen in the pre-trained data.
• Score: 23.094280494500214
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Humanoid loco-manipulation requires coordinated high-level motion plans with stable, low-level whole-body execution under complex robot-environment dynamics and long-horizon tasks. While diffusion policies (DPs) show promise for learning from demonstrations, deploying them on humanoids poses critical challenges: the motion planner trained offline is decoupled from the low-level controller, leading to poor command tracking, compounding distribution shift, and task failures. The common approach of scaling demonstration data is prohibitively expensive for high-dimensional humanoid systems. To address this challenge, we present REFINE-DP (REinforcement learning FINE-tuning of Diffusion Policy), a hierarchical framework that jointly optimizes a DP high-level planner and an RL-based low-level loco-manipulation controller. The DP is fine-tuned via a PPO-based diffusion policy gradient to improve task success rate, while the controller is simultaneously updated to accurately track the planner's evolving command distribution, reducing the distributional mismatch that degrades motion quality. We validate REFINE-DP on a humanoid robot performing loco-manipulation tasks, including door traversal and long-horizon object transport. REFINE-DP achieves an over $90\%$ success rate in simulation, even in out-of-distribution cases not seen in the pre-trained data, and enables smooth autonomous task execution in real-world dynamic environments. Our proposed method substantially outperforms pre-trained DP baselines and demonstrates that RL fine-tuning is key to reliable humanoid loco-manipulation. https://refine-dp.github.io/REFINE-DP/

Related papers

• HAF: Adapting Generalist VLAs to Humanoid Whole-Body Loco-manipulation via Hierarchical Action Flow and Spectral Latent RL [56.014913786737104]
  We introduce HAF (Humanoid Adaptation Framework) to transfer off-the-shelf generalist VLA foundation models to humanoid whole-body loco-manipulation.<n> HAF-VLA is a hierarchical action-flow generator built on a pretrained flow-matching VLA.<n>HaF-Steer is a latent offline-to-online RL pipeline that leverages flow-matching invertibility and DCT-based dimensionality reduction.
  arXiv  Detail & Related papers  (2026-08-17T17:22:33Z)
• PAC-DP: PAC-Bayesian Diffusion Policy Learning [13.10166905442267]
  Diffusion Policies (DPs) are able to perform complex manipulation tasks.<n>We propose PAC-DP, an approach that increases the performance of DPs in robotic manipulation tasks.
  arXiv  Detail & Related papers  (2026-07-27T11:43:49Z)
• GHOST: Hierarchical Sub-Goal Policies for Generalizing Robot Manipulation [54.305781492241785]
  We present a framework for learning visuomotor manipulation policies that generalize beyond the training distribution.<n>GHOST factorizes control into (i) a high-level policy that predicts the next sub-goal as a distribution over 3D end-effector poses from multi-view RGB-D observations.<n>Across a suite of tasks, this hierarchical factorization consistently improves performance and robustness compared to a flat Diffusion Policy.
  arXiv  Detail & Related papers  (2026-06-08T18:08:01Z)
• Set-Supervised Diffusion Policy: Learning Action-Chunking Diffusion through Corrections [25.961145214027255]
  Diffusion policies provide paired supervision in the form of the robot's undesired actions and the human teacher's corrective actions.<n>We propose Set-Supervised Diffusion Policy (SDP), a novel learning framework that utilizes contrastive action-chunk data to train diffusion policies from human corrections.<n>SDP consistently improves policy performance, with particularly strong gains in robustness to noisy data.
  arXiv  Detail & Related papers  (2026-06-01T08:14:38Z)
• Closed-Loop Action Chunks with Dynamic Corrections for Training-Free Diffusion Policy [52.106797722292896]
  We present DCDP, a Dynamic Closed-Loop Diffusion Policy framework that integrates chunk-based action generation with real-time correction.<n>In dynamic PushT simulations, DCDP improves adaptability by 19% without retraining while requiring only 5% additional computation.
  arXiv  Detail & Related papers  (2026-03-02T15:04:18Z)
• HiWET: Hierarchical World-Frame End-Effector Tracking for Long-Horizon Humanoid Loco-Manipulation [43.43128572722804]
  We reformulate the problem as world-frame end-effector tracking for humanoid loco-manipulation.<n>We propose HiWET, a hierarchical reinforcement learning framework that decouples global reasoning from dynamic execution.<n>We show that HiWET achieves precise and stable end-effector tracking in long-horizon world-frame tasks.
  arXiv  Detail & Related papers  (2026-02-06T03:11:56Z)
• DDP-WM: Disentangled Dynamics Prediction for Efficient World Models [79.53092337527382]
  We introduce DDP-WM, a novel world model centered on the principle of Disentangled Dynamics Prediction.<n>DDP-WM realizes this decomposition through an architecture that integrates efficient historical processing with dynamic localization.<n>Experiments demonstrate that DDP-WM achieves significant efficiency and performance across diverse tasks.
  arXiv  Detail & Related papers  (2026-02-02T08:04:25Z)
• Push Smarter, Not Harder: Hierarchical RL-Diffusion Policy for Efficient Nonprehensile Manipulation [8.7216199131049]
  HeRD is a hierarchical reinforcement learning-diffusion policy that decomposes pushing tasks into two levels: high-level goal selection and low-level trajectory generation.<n>We employ a high-level reinforcement learning agent to select intermediate spatial goals, and a low-level goal-conditioned diffusion model to generate feasible, efficient trajectories to reach them.<n>Our results suggest that hierarchical control with generative low-level planning is a promising direction for scalable, goal-directed nonprehensile manipulation.
  arXiv  Detail & Related papers  (2025-12-10T21:40:22Z)
• Learning-based Cooperative Robotic Paper Wrapping: A Unified Control Policy with Residual Force Control [11.21445976755808]
  We propose a learning-based framework that integrates a high-level task planner powered by a large language model with a low-level hybrid imitation learning and reinforcement learning policy.<n>At its core is a Sub-task Aware Robotic Transformer (START) that learns a unified policy from human demonstrations.<n>We show that the unified transformer-based policy reduces the need for specialized models, allows controlled human supervision, and effectively bridges high-level intent with the fine-grained force control required for deformable object manipulation.
  arXiv  Detail & Related papers  (2025-11-05T04:55:35Z)
• One-Step Diffusion Policy: Fast Visuomotor Policies via Diffusion Distillation [80.71541671907426]
  OneStep Diffusion Policy (OneDP) is a novel approach that distills knowledge from pre-trained diffusion policies into a single-step action generator. OneDP significantly accelerates response times for robotic control tasks.
  arXiv  Detail & Related papers  (2024-10-28T17:54:31Z)
• Latent Weight Diffusion: Generating reactive policies instead of trajectories [12.270795590154489]
  We propose Latent Weight Diffusion to generate closed-loop policies for robotic tasks.<n>LWD has higher success rates than Diffusion Policy when the action horizon is longer.<n>LWD achieves multitask performance comparable to DP while requiring just 1/45th of the inference-time FLOPS.
  arXiv  Detail & Related papers  (2024-10-17T21:30:29Z)
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model [119.65409513119963]
  We introduce a new parameterization of the reward model in RLHF that enables extraction of the corresponding optimal policy in closed form. The resulting algorithm, which we call Direct Preference Optimization (DPO), is stable, performant, and computationally lightweight. Our experiments show that DPO can fine-tune LMs to align with human preferences as well as or better than existing methods.
  arXiv  Detail & Related papers  (2023-05-29T17:57:46Z)
• Learning to Simulate Self-Driven Particles System with Coordinated Policy Optimization [36.113662645486755]
  Self-Driven Particles (SDP) describe a category of multi-agent systems common in everyday life, such as flocking birds and traffic flows. Previous multi-agent reinforcement learning (MARL) methods define the agents to be teammates or enemies before hand. In this work we develop a novel MARL method called Coordinated Policy Optimization (CoPO), which incorporates social psychology principle to learn neural controller for SDP.
  arXiv  Detail & Related papers  (2021-10-26T16:20:23Z)
• Modular Deep Reinforcement Learning for Continuous Motion Planning with Temporal Logic [59.94347858883343]
  This paper investigates the motion planning of autonomous dynamical systems modeled by Markov decision processes (MDP) The novelty is to design an embedded product MDP (EP-MDP) between the LDGBA and the MDP. The proposed LDGBA-based reward shaping and discounting schemes for the model-free reinforcement learning (RL) only depend on the EP-MDP states.
  arXiv  Detail & Related papers  (2021-02-24T01:11:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.