Iterative On-Policy Refinement of Hierarchical Diffusion Policies for Language-Conditioned Manipulation

• URL: http://arxiv.org/abs/2603.05291v1
• Date: Thu, 05 Mar 2026 15:34:43 GMT
• Title: Iterative On-Policy Refinement of Hierarchical Diffusion Policies for Language-Conditioned Manipulation
• Abstract summary: HD-ExpIt is a framework for iterative fine-tuning of hierarchical diffusion policies via environment feedback.<n>It significantly improves hierarchical policies trained solely on offline data.
• Score: 6.115550515200226
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Hierarchical policies for language-conditioned manipulation decompose tasks into subgoals, where a high-level planner guides a low-level controller. However, these hierarchical agents often fail because the planner generates subgoals without considering the actual limitations of the controller. Existing solutions attempt to bridge this gap via intermediate modules or shared representations, but they remain limited by their reliance on fixed offline datasets. We propose HD-ExpIt, a framework for iterative fine-tuning of hierarchical diffusion policies via environment feedback. HD-ExpIt organizes training into a self-reinforcing cycle: it utilizes diffusion-based planning to autonomously discover successful behaviors, which are then distilled back into the hierarchical policy. This loop enables both components to improve while implicitly grounding the planner in the controller's actual capabilities without requiring explicit proxy models. Empirically, HD-ExpIt significantly improves hierarchical policies trained solely on offline data, achieving state-of-the-art performance on the long-horizon CALVIN benchmark among methods trained from scratch.

Related papers

• GHOST: Hierarchical Sub-Goal Policies for Generalizing Robot Manipulation [54.305781492241785]
  We present a framework for learning visuomotor manipulation policies that generalize beyond the training distribution.<n>GHOST factorizes control into (i) a high-level policy that predicts the next sub-goal as a distribution over 3D end-effector poses from multi-view RGB-D observations.<n>Across a suite of tasks, this hierarchical factorization consistently improves performance and robustness compared to a flat Diffusion Policy.
  arXiv  Detail & Related papers  (2026-06-08T18:08:01Z)
• Switching Successor Measures for Hierarchical Zero-shot Reinforcement Learning [49.24483784910263]
  We introduce switching successor measures, an extension of successor measures that enables hierarchical control in reinforcement learning.<n>We show that switching successor measures arise naturally from classical successor measures while preserving their structure.<n>Experiments on both goal-conditioned and general reward-based tasks show that FB $$-Switch improves over non-hierarchical baselines.
  arXiv  Detail & Related papers  (2026-05-13T08:58:33Z)
• Push Smarter, Not Harder: Hierarchical RL-Diffusion Policy for Efficient Nonprehensile Manipulation [8.7216199131049]
  HeRD is a hierarchical reinforcement learning-diffusion policy that decomposes pushing tasks into two levels: high-level goal selection and low-level trajectory generation.<n>We employ a high-level reinforcement learning agent to select intermediate spatial goals, and a low-level goal-conditioned diffusion model to generate feasible, efficient trajectories to reach them.<n>Our results suggest that hierarchical control with generative low-level planning is a promising direction for scalable, goal-directed nonprehensile manipulation.
  arXiv  Detail & Related papers  (2025-12-10T21:40:22Z)
• Embodied Long Horizon Manipulation with Closed-loop Code Generation and Incremental Few-shot Adaptation [12.077740860502878]
  Embodied long-horizon manipulation requires robotic systems to process multimodal inputs-such as vision and natural language-and translate them into executable actions.<n>Recent methods have explored using large language models (LLMs) as high-level planners that decompose tasks into subtasks using natural language and guide pretrained low-level controllers.<n>Our framework achieves state-of-the-art performance on 30+ diverse seen and unseen long-horizon tasks across LoHoRavens, CALVIN, Franka Kitchen, and cluttered real-world settings.
  arXiv  Detail & Related papers  (2025-03-27T20:32:58Z)
• Direct Preference Optimization for Primitive-Enabled Hierarchical Reinforcement Learning [75.9729413703531]
  DIPPER is a novel HRL framework that formulates hierarchical policy learning as a bi-level optimization problem.<n>We show that DIPPER achieves up to 40% improvement over state-of-the-art baselines in sparse reward scenarios.
  arXiv  Detail & Related papers  (2024-11-01T04:58:40Z)
• Offline Hierarchical Reinforcement Learning via Inverse Optimization [23.664330010602708]
  OHIO is a framework for offline reinforcement learning of hierarchical policies.<n>We show it substantially outperforms end-to-end RL methods and improves robustness.
  arXiv  Detail & Related papers  (2024-10-10T14:00:21Z)
• AD-H: Autonomous Driving with Hierarchical Agents [64.49185157446297]
  We propose to connect high-level instructions and low-level control signals with mid-level language-driven commands. We implement this idea through a hierarchical multi-agent driving system named AD-H.
  arXiv  Detail & Related papers  (2024-06-05T17:25:46Z)
• Feudal Graph Reinforcement Learning [18.069747511100132]
  Graph-based representations and message-passing modular policies constitute prominent approaches to tackling composable control problems in reinforcement learning (RL)<n>We propose a novel methodology, named Feudal Graph Reinforcement Learning (FGRL), that addresses such challenges by relying on hierarchical RL and a pyramidal message-passing architecture.<n>In particular, FGRL defines a hierarchy of policies where high-level commands are propagated from the top of the hierarchy down through a layered graph structure.
  arXiv  Detail & Related papers  (2023-04-11T09:51:13Z)
• Distributed-Training-and-Execution Multi-Agent Reinforcement Learning for Power Control in HetNet [48.96004919910818]
  We propose a multi-agent deep reinforcement learning (MADRL) based power control scheme for the HetNet. To promote cooperation among agents, we develop a penalty-based Q learning (PQL) algorithm for MADRL systems. In this way, an agent's policy can be learned by other agents more easily, resulting in a more efficient collaboration process.
  arXiv  Detail & Related papers  (2022-12-15T17:01:56Z)
• Planning to Practice: Efficient Online Fine-Tuning by Composing Goals in Latent Space [76.46113138484947]
  General-purpose robots require diverse repertoires of behaviors to complete challenging tasks in real-world unstructured environments. To address this issue, goal-conditioned reinforcement learning aims to acquire policies that can reach goals for a wide range of tasks on command. We propose Planning to Practice, a method that makes it practical to train goal-conditioned policies for long-horizon tasks.
  arXiv  Detail & Related papers  (2022-05-17T06:58:17Z)
• Learning Functionally Decomposed Hierarchies for Continuous Control Tasks with Path Planning [36.050432925402845]
  We present HiDe, a novel hierarchical reinforcement learning architecture that successfully solves long horizon control tasks. We experimentally show that our method generalizes across unseen test environments and can scale to 3x horizon length compared to both learning and non-learning based methods.
  arXiv  Detail & Related papers  (2020-02-14T10:19:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.