Gated Q-learning: Add Off-Policy Bias to Taste

• URL: http://arxiv.org/abs/2607.28916v1
• Date: Fri, 31 Jul 2026 00:32:46 GMT
• Title: Gated Q-learning: Add Off-Policy Bias to Taste
• Abstract summary: Managing off-policy bias in Q-learning remains a fundamental challenge.<n>Gated Q-learning offers a simple alternative to importance sampling while enabling customization of the effective multistep horizon.<n> Empirical evaluations verify that intermediate gating safely enables longer credit-assignment horizons.
• Score: 2.7074235008521246
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multistep credit assignment is critical for sample-efficient reinforcement learning, yet managing off-policy bias in Q-learning remains a fundamental challenge. For 30 years, practitioners have been limited to a binary choice: eliminate the bias at the cost of severely truncated eligibility traces (Watkins' Q($λ$)), or ignore the bias to learn faster while injecting detrimental errors into the value estimates (Peng's Q($λ$)). Modern off-policy estimators fail to resolve this tension, as importance-sampling ratios collapse under Q-learning's greedy target policy. We introduce Gated Q-learning, a novel algorithmic framework that ends this dilemma by smoothly interpolating between the two historical extremes. Rather than relying on importance sampling, our approach employs a continuous, state-action-dependent gating mechanism to selectively attenuate eligibility traces in an exploration-aware manner. We provide a rigorous theoretical foundation for this mechanism, proving that the expected operator remains a contraction mapping and deriving its exact fixed point. Empirical evaluations verify that intermediate gating safely enables longer credit-assignment horizons, yielding faster initial learning than either extreme. Gated Q-learning offers a simple alternative to importance sampling while enabling customization of the effective multistep horizon and the amount of off-policy bias in Q-learning agents.

Related papers

• Q-learning with Adjoint Matching [58.78551025170267]
  We propose Q-learning with Adjoint Matching (QAM), a novel TD-based reinforcement learning (RL) algorithm.<n>QAM sidesteps two challenges by leveraging adjoint matching, a recently proposed technique in generative modeling.<n>It consistently outperforms prior approaches on hard, sparse reward tasks in both offline and offline-to-online RL.
  arXiv  Detail & Related papers  (2026-01-20T18:45:34Z)
• Is Q-learning an Ill-posed Problem? [2.4424095531386234]
  This paper investigates the instability of Q-learning in continuous environments.<n>We show that even in relatively simple benchmarks, the fundamental task of Q-learning can be inherently ill-posed and prone to failure.
  arXiv  Detail & Related papers  (2025-02-20T08:42:30Z)
• Regularized Q-learning through Robust Averaging [3.4354636842203026]
  We propose a new Q-learning variant, called 2RA Q-learning, that addresses some weaknesses of existing Q-learning methods in a principled manner. One such weakness is an underlying estimation bias which cannot be controlled and often results in poor performance. We show that 2RA Q-learning converges to the optimal policy and analyze its theoretical mean-squared error.
  arXiv  Detail & Related papers  (2024-05-03T15:57:26Z)
• Projected Off-Policy Q-Learning (POP-QL) for Stabilizing Offline Reinforcement Learning [57.83919813698673]
  Projected Off-Policy Q-Learning (POP-QL) is a novel actor-critic algorithm that simultaneously reweights off-policy samples and constrains the policy to prevent divergence and reduce value-approximation error. In our experiments, POP-QL not only shows competitive performance on standard benchmarks, but also out-performs competing methods in tasks where the data-collection policy is significantly sub-optimal.
  arXiv  Detail & Related papers  (2023-11-25T00:30:58Z)
• Trajectory-Aware Eligibility Traces for Off-Policy Reinforcement Learning [44.50394347326546]
  Off-policy learning from multistep returns is crucial for sample-efficient reinforcement learning. Off-policy bias is corrected in a per-decision manner, but once a trace has been fully cut, the effect cannot be reversed. We propose a multistep operator that can express both per-decision and trajectory-aware methods.
  arXiv  Detail & Related papers  (2023-01-26T18:57:41Z)
• Balanced Q-learning: Combining the Influence of Optimistic and Pessimistic Targets [74.04426767769785]
  We show that specific types of biases may be preferable, depending on the scenario. We design a novel reinforcement learning algorithm, Balanced Q-learning, in which the target is modified to be a convex combination of a pessimistic and an optimistic term.
  arXiv  Detail & Related papers  (2021-11-03T07:30:19Z)
• Temporal-Difference Value Estimation via Uncertainty-Guided Soft Updates [110.92598350897192]
  Q-Learning has proven effective at learning a policy to perform control tasks. estimation noise becomes a bias after the max operator in the policy improvement step. We present Unbiased Soft Q-Learning (UQL), which extends the work of EQL from two action, finite state spaces to multi-action, infinite state Markov Decision Processes.
  arXiv  Detail & Related papers  (2021-10-28T00:07:19Z)
• On the Estimation Bias in Double Q-Learning [20.856485777692594]
  Double Q-learning is not fully unbiased and suffers from underestimation bias. We show that such underestimation bias may lead to multiple non-optimal fixed points under an approximated Bellman operator. We propose a simple but effective approach as a partial fix for the underestimation bias in double Q-learning.
  arXiv  Detail & Related papers  (2021-09-29T13:41:24Z)
• Cross Learning in Deep Q-Networks [82.20059754270302]
  We propose a novel cross Q-learning algorithm, aim at alleviating the well-known overestimation problem in value-based reinforcement learning methods. Our algorithm builds on double Q-learning, by maintaining a set of parallel models and estimate the Q-value based on a randomly selected network.
  arXiv  Detail & Related papers  (2020-09-29T04:58:17Z)
• ConQUR: Mitigating Delusional Bias in Deep Q-learning [45.21332566843924]
  Delusional bias is a fundamental source of error in approximate Q-learning. We develop efficient methods to mitigate delusional bias by training Q-approximators with labels that are "consistent" with the underlying greedy policy class.
  arXiv  Detail & Related papers  (2020-02-27T19:22:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.