Efficient Hierarchical Implicit Flow Q-learning for Offline Goal-conditioned Reinforcement Learning

• URL: http://arxiv.org/abs/2604.08960v1
• Date: Fri, 10 Apr 2026 05:04:29 GMT
• Title: Efficient Hierarchical Implicit Flow Q-learning for Offline Goal-conditioned Reinforcement Learning
• Abstract summary: offline goal-conditioned reinforcement learning (GCRL) is a practical reinforcement learning paradigm that aims to learn goal-conditioned policies from reward-free offline data.<n>We propose the goal-conditioned mean flow policy, which introduces an average velocity field into hierarchical policy modeling.<n>We show that our method achieves strong performance across both state-based and pixel-based tasks in the OGBench benchmark.
• Score: 10.729712481687628
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Offline goal-conditioned reinforcement learning (GCRL) is a practical reinforcement learning paradigm that aims to learn goal-conditioned policies from reward-free offline data. Despite recent advances in hierarchical architectures such as HIQL, long-horizon control in offline GCRL remains challenging due to the limited expressiveness of Gaussian policies and the inability of high-level policies to generate effective subgoals. To address these limitations, we propose the goal-conditioned mean flow policy, which introduces an average velocity field into hierarchical policy modeling for offline GCRL. Specifically, the mean flow policy captures complex target distributions for both high-level and low-level policies through a learned average velocity field, enabling efficient action generation via one-step sampling. Furthermore, considering the insufficiency of goal representation, we introduce a LeJEPA loss that repels goal representation embeddings during training, thereby encouraging more discriminative representations and improving generalization. Experimental results show that our method achieves strong performance across both state-based and pixel-based tasks in the OGBench benchmark.

Related papers

• Test-Time Gradient Guidance of Flow Policies in Reinforcement Learning [50.738952715864116]
  Expressive continuous control policies form the backbone of advances in scaling imitation learning for simulated and real robot control.<n>We propose QGF (Q-Guided Flow), an RL algorithm that performs policy optimization entirely at test time.<n> Empirically, QGF outperforms prior test-time RL methods on single-task and goal-conditioned offline RL benchmarks.
  arXiv  Detail & Related papers  (2026-06-09T16:45:57Z)
• Mean Flow Policy with Instantaneous Velocity Constraint for One-step Action Generation [65.13627721310613]
  Mean velocity policy (MVP) is a new generative policy function that models the mean velocity field to achieve the fastest one-step action generation.<n>MVP achieves state-of-the-art success rates across several challenging robotic manipulation tasks from Robomimic and OGBench.
  arXiv  Detail & Related papers  (2026-02-14T14:44:06Z)
• Data-Efficient Hierarchical Goal-Conditioned Reinforcement Learning via Normalizing Flows [0.3499870393443268]
  Normalizing flow-based hierarchical implicit Q-learning (NF-HIQL)<n>New theoretical guarantees are derived, including explicit KL-divergence bounds for Real-valued non-volume preserving (RealNVP) policies.<n> NF-HIQL consistently outperforms prior goal-conditioned and hierarchical baselines.
  arXiv  Detail & Related papers  (2026-02-11T18:54:48Z)
• Polychromic Objectives for Reinforcement Learning [63.37185057794815]
  Reinforcement learning fine-tuning (RLFT) is a dominant paradigm for improving pretrained policies for downstream tasks.<n>We introduce an objective for policy methods that explicitly enforces the exploration and refinement of diverse generations.<n>We show how proximal policy optimization (PPO) can be adapted to optimize this objective.
  arXiv  Detail & Related papers  (2025-09-29T19:32:11Z)
• Hierarchical Reinforcement Learning with Uncertainty-Guided Diffusional Subgoals [12.894271401094615]
  A key challenge in HRL is that the low-level policy changes over time, making it difficult for the high-level policy to generate effective subgoals.<n>We propose an approach that trains a conditional diffusion model regularized by a Gaussian Process (GP) prior to generate a complex variety of subgoals.<n>Building on this framework, we develop a strategy that selects subgoals from both the diffusion policy and GP's predictive mean.
  arXiv  Detail & Related papers  (2025-05-27T20:38:44Z)
• Flattening Hierarchies with Policy Bootstrapping [5.528896840956629]
  We introduce an algorithm to train a flat (non-hierarchical) goal-conditioned policy by bootstrapping on subgoal-conditioned policies with advantage-weighted importance sampling.<n>Our approach eliminates the need for a generative model over the (sub)goal space, which we find is key for scaling to high-dimensional control in large state spaces.
  arXiv  Detail & Related papers  (2025-05-20T23:31:30Z)
• Option-aware Temporally Abstracted Value for Offline Goal-Conditioned Reinforcement Learning [19.341894845618445]
  offline goal-conditioned reinforcement learning (GCRL) offers a practical learning paradigm in which goal-reaching policies are trained from abundant state-action trajectory datasets.<n>In this paper, we propose a simple yet effective solution: Option-aware Temporally Abstracted value learning, dubbed OTA, which incorporates temporal abstraction into the temporal-difference learning process.<n>We experimentally show that the high-level policy learned using OTA achieves strong performance on complex tasks from OGBench, a recently proposed offline GCRL benchmark.
  arXiv  Detail & Related papers  (2025-05-19T05:51:11Z)
• Direct Preference Optimization for Primitive-Enabled Hierarchical Reinforcement Learning [75.9729413703531]
  DIPPER is a novel HRL framework that formulates hierarchical policy learning as a bi-level optimization problem.<n>We show that DIPPER achieves up to 40% improvement over state-of-the-art baselines in sparse reward scenarios.
  arXiv  Detail & Related papers  (2024-11-01T04:58:40Z)
• Offline Policy Learning via Skill-step Abstraction for Long-horizon Goal-Conditioned Tasks [7.122367852177223]
  We present an offline GC policy learning' framework tailored for tackling long-horizon GC tasks. In the framework, a GC policy is progressively learned offline in conjunction with the incremental modeling of skill-step abstractions on the data. We demonstrate the superiority and efficiency of our GLvSA framework in adapting GC policies to a wide range of long-horizon goals.
  arXiv  Detail & Related papers  (2024-08-21T03:05:06Z)
• Imitating Graph-Based Planning with Goal-Conditioned Policies [72.61631088613048]
  We present a self-imitation scheme which distills a subgoal-conditioned policy into the target-goal-conditioned policy. We empirically show that our method can significantly boost the sample-efficiency of the existing goal-conditioned RL methods.
  arXiv  Detail & Related papers  (2023-03-20T14:51:10Z)
• Conservative Q-Learning for Offline Reinforcement Learning [106.05582605650932]
  We show that CQL substantially outperforms existing offline RL methods, often learning policies that attain 2-5 times higher final return. We theoretically show that CQL produces a lower bound on the value of the current policy and that it can be incorporated into a policy learning procedure with theoretical improvement guarantees.
  arXiv  Detail & Related papers  (2020-06-08T17:53:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.