Offline RL with Hierarchical Action Chunking

• URL: http://arxiv.org/abs/2607.20834v1
• Date: Thu, 23 Jul 2026 01:48:46 GMT
• Title: Offline RL with Hierarchical Action Chunking
• Abstract summary: Hierarchical Implicit Q-Chunking (HiQC) is an offline goal-conditioned RL algorithm that combines high-level latent planning with low-level action chunking.<n>HiQC achieves the highest aggregate performance among the compared methods on the OGBench suite.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Offline goal-conditioned reinforcement learning (RL) holds the promise of learning general-purpose policies from static datasets. However, scaling these methods to long-horizon tasks remains a challenge due to the curse of horizon, where value estimation errors can compound through long chains of bootstrapped Bellman backups. Existing hierarchical approaches mitigate this by decomposing tasks into subgoals, yet they often rely on low-level controllers that suffer from myopic execution and biased value estimates. In this work, we propose Hierarchical Implicit Q-Chunking (HiQC), an offline goal-conditioned RL algorithm that combines high-level latent planning with low-level action chunking. By conditioning the low-level critic on temporally extended action sequences, HiQC enables unbiased k-step value backups, compressing the horizon at both the planning and execution levels. We theoretically demonstrate that this dual decomposition results in a tighter bound on value error under a bounded per-backup error model compared to standard hierarchy or flat chunking alone. Empirically, HiQC achieves the highest aggregate performance among the compared methods on the OGBench suite, with its largest gains on long-horizon navigation tasks such as humanoid-giant.

Related papers

• Adaptive Coarse-to-Fine Subgoal Refinement for Long-Horizon Offline Goal-Conditioned Reinforcement Learning [16.15046948057208]
  We propose a fully offline goal-conditioned reinforcement learning framework that adaptively refines distant goals before execution.<n>Experiments on OGBench show substantial gains on several long-horizon tasks, with ablations validating the proposed refinement and stopping mechanisms.
  arXiv  Detail & Related papers  (2026-05-27T08:17:01Z)
• Long-Horizon Manipulation via Trace-Conditioned VLA Planning [55.80061850746898]
  LoHo-Manip is a framework that scales short-horizon VLA execution to long-horizon instruction following via a dedicated task-management VLM.<n>LoHo-Manip predicts a progress-aware remaining plan that combines a subtask sequence with an explicit done + remaining split as lightweight language memory.<n>The executor VLA is adapted to condition on the rendered trace, thereby turning long-horizon decision-making into repeated local control.
  arXiv  Detail & Related papers  (2026-04-23T17:59:04Z)
• HiPER: Hierarchical Reinforcement Learning with Explicit Credit Assignment for Large Language Model Agents [36.77027704958893]
  HiPER is a novel Hierarchical Plan-Execute RL framework that separates high-level planning from low-level execution.<n>HiPER achieves state-of-the-art performance on challenging interactive benchmarks, reaching 97.4% success on ALFWorld and 83.3% on WebShop with Qwen2.5-7B-Instruct.
  arXiv  Detail & Related papers  (2026-02-18T03:31:34Z)
• Chain-of-Goals Hierarchical Policy for Long-Horizon Offline Goal-Conditioned RL [25.40364932514488]
  We propose a novel framework that reformulates hierarchical decision-making as autoregressive sequence modeling.<n>CoGHP consistently outperforms strong offline baselines, demonstrating improved performance on long-horizon tasks.
  arXiv  Detail & Related papers  (2026-02-03T11:11:03Z)
• ReCAP: Recursive Context-Aware Reasoning and Planning for Large Language Model Agents [61.51091799997476]
  We introduce ReCAP (Recursive Context-Aware Reasoning and Planning), a hierarchical framework with shared context for reasoning and planning in large language models (LLMs)<n>ReCAP combines three key mechanisms: plan-ahead decomposition, structured re-injection of parent plans, and memory-efficient execution.<n>Experiments demonstrate that ReCAP substantially improves subgoal alignment and success rates on various long-horizon reasoning benchmarks.
  arXiv  Detail & Related papers  (2025-10-27T20:03:55Z)
• Strict Subgoal Execution: Reliable Long-Horizon Planning in Hierarchical Reinforcement Learning [5.274804664403783]
  Strict Subgoal Execution (SSE) is a graph-based hierarchical RL framework that enforces single-step subgoal reachability.<n>We show that SSE consistently outperforms existing goal-conditioned RL and hierarchical RL approaches in both efficiency and success rate.
  arXiv  Detail & Related papers  (2025-06-26T06:35:42Z)
• PlanDQ: Hierarchical Plan Orchestration via D-Conductor and Q-Performer [47.924941959320996]
  We propose a hierarchical planner designed for offline RL called PlanDQ. PlanDQ incorporates a diffusion-based planner at the high level, named D-Conductor, which guides the low-level policy through sub-goals. At the low level, we used a Q-learning based approach called the Q-Performer to accomplish these sub-goals.
  arXiv  Detail & Related papers  (2024-06-10T20:59:53Z)
• HIQL: Offline Goal-Conditioned RL with Latent States as Actions [81.67963770528753]
  We propose a hierarchical algorithm for goal-conditioned RL from offline data. We show how this hierarchical decomposition makes our method robust to noise in the estimated value function. Our method can solve long-horizon tasks that stymie prior methods, can scale to high-dimensional image observations, and can readily make use of action-free data.
  arXiv  Detail & Related papers  (2023-07-22T00:17:36Z)
• Conservative Q-Learning for Offline Reinforcement Learning [106.05582605650932]
  We show that CQL substantially outperforms existing offline RL methods, often learning policies that attain 2-5 times higher final return. We theoretically show that CQL produces a lower bound on the value of the current policy and that it can be incorporated into a policy learning procedure with theoretical improvement guarantees.
  arXiv  Detail & Related papers  (2020-06-08T17:53:42Z)
• Hierarchical Reinforcement Learning as a Model of Human Task Interleaving [60.95424607008241]
  We develop a hierarchical model of supervisory control driven by reinforcement learning. The model reproduces known empirical effects of task interleaving. The results support hierarchical RL as a plausible model of task interleaving.
  arXiv  Detail & Related papers  (2020-01-04T17:53:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.