Chunk-Guided Q-Learning

• URL: http://arxiv.org/abs/2603.13971v1
• Date: Sat, 14 Mar 2026 14:54:02 GMT
• Title: Chunk-Guided Q-Learning
• Abstract summary: Chunk-Guided Q-Learning (CGQ) guides a fine-grained single-step critic by regularizing it toward a chunk-based critic trained using temporally extended backups.<n>We show that CGQ attains tighter critic optimality bounds than either single-step or action-chunked TD learning alone.<n> Empirically, CGQ achieves strong performance on challenging long-horizon OGBench tasks, often outperforming both single-step and action-chunked methods.
• Score: 9.48300389412777
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In offline reinforcement learning (RL), single-step temporal-difference (TD) learning can suffer from bootstrapping error accumulation over long horizons. Action-chunked TD methods mitigate this by backing up over multiple steps, but can introduce suboptimality by restricting the policy class to open-loop action sequences. To resolve this trade-off, we present Chunk-Guided Q-Learning (CGQ), a single-step TD algorithm that guides a fine-grained single-step critic by regularizing it toward a chunk-based critic trained using temporally extended backups. This reduces compounding error while preserving fine-grained value propagation. We theoretically show that CGQ attains tighter critic optimality bounds than either single-step or action-chunked TD learning alone. Empirically, CGQ achieves strong performance on challenging long-horizon OGBench tasks, often outperforming both single-step and action-chunked methods.

Related papers

• Long-Horizon Q-Learning: Accurate Value Learning via n-Step Inequalities [46.51268762094611]
  We propose a principled backstop against compounding error when learning the optimal action-value function.<n>Long-horizon Q-learning (LQL) builds on a prior optimality tightening observation.<n>LQL consistently outperforms both 1-step TD and n-step TD learning at similar runtime.
  arXiv  Detail & Related papers  (2026-05-07T07:47:55Z)
• What Does Flow Matching Bring To TD Learning? [28.717975688380488]
  Flow matching can be effective for scalar Q-value function estimation in reinforcement learning (RL)<n>We show that their success is not explained by distributional RL, as explicitly modeling return distributions can reduce performance.<n>We argue that the use of integration for reading out values and dense velocity supervision at each step of this integration process for training improves TD learning via two mechanisms.
  arXiv  Detail & Related papers  (2026-03-04T17:51:30Z)
• Q-learning with Adjoint Matching [58.78551025170267]
  We propose Q-learning with Adjoint Matching (QAM), a novel TD-based reinforcement learning (RL) algorithm.<n>QAM sidesteps two challenges by leveraging adjoint matching, a recently proposed technique in generative modeling.<n>It consistently outperforms prior approaches on hard, sparse reward tasks in both offline and offline-to-online RL.
  arXiv  Detail & Related papers  (2026-01-20T18:45:34Z)
• Decoupled Q-Chunking [63.864222078287575]
  We use chunked critics to estimate the value of short action sequences ("chunks") rather than individual actions, speeding up value backup.<n>Our key insight is to decouple the chunk length of the critic from that of the policy, allowing the policy to operate over shorter action chunks.<n>This design retains the benefits of multi-step value propagation while sidestepping both the open-loop sub-optimality and the difficulty of learning action chunking policies for long action chunks.
  arXiv  Detail & Related papers  (2025-12-11T18:52:51Z)
• Time-Scale Separation in Q-Learning: Extending TD($\triangle$) for Action-Value Function Decomposition [0.0]
  This paper introduces Q($Delta$)-Learning, an extension of TD($Delta$) for the Q-Learning framework. TD($Delta$) facilitates efficient learning over several time scales by breaking the Q($Delta$)-function into distinct discount factors. We demonstrate through theoretical analysis and practical evaluations on standard benchmarks like Atari that Q($Delta$)-Learning surpasses conventional Q-Learning and TD learning methods.
  arXiv  Detail & Related papers  (2024-11-21T11:03:07Z)
• SLCA++: Unleash the Power of Sequential Fine-tuning for Continual Learning with Pre-training [68.7896349660824]
  We present an in-depth analysis of the progressive overfitting problem from the lens of Seq FT. Considering that the overly fast representation learning and the biased classification layer constitute this particular problem, we introduce the advanced Slow Learner with Alignment (S++) framework. Our approach involves a Slow Learner to selectively reduce the learning rate of backbone parameters, and a Alignment to align the disjoint classification layers in a post-hoc fashion.
  arXiv  Detail & Related papers  (2024-08-15T17:50:07Z)
• Efficient Offline Reinforcement Learning: The Critic is Critical [5.916429671763282]
  Off-policy reinforcement learning provides a promising approach for improving performance beyond supervised approaches. We propose a best-of-both approach by first learning the behavior policy and critic with supervised learning, before improving with off-policy reinforcement learning.
  arXiv  Detail & Related papers  (2024-06-19T09:16:38Z)
• Solving Continuous Control via Q-learning [54.05120662838286]
  We show that a simple modification of deep Q-learning largely alleviates issues with actor-critic methods. By combining bang-bang action discretization with value decomposition, framing single-agent control as cooperative multi-agent reinforcement learning (MARL), this simple critic-only approach matches performance of state-of-the-art continuous actor-critic methods.
  arXiv  Detail & Related papers  (2022-10-22T22:55:50Z)
• Simultaneous Double Q-learning with Conservative Advantage Learning for Actor-Critic Methods [133.85604983925282]
  We propose Simultaneous Double Q-learning with Conservative Advantage Learning (SDQ-CAL) Our algorithm realizes less biased value estimation and achieves state-of-the-art performance in a range of continuous control benchmark tasks.
  arXiv  Detail & Related papers  (2022-05-08T09:17:16Z)
• GRAC: Self-Guided and Self-Regularized Actor-Critic [24.268453994605512]
  We propose a self-regularized TD-learning method to address divergence without requiring a target network. We also propose a self-guided policy improvement method by combining policy-gradient with zero-order optimization. This makes learning more robust to local noise in the Q function approximation and guides the updates of our actor network. We evaluate GRAC on the suite of OpenAI gym tasks, achieving or outperforming state of the art in every environment tested.
  arXiv  Detail & Related papers  (2020-09-18T17:58:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.