Heavy-Ball Q-Learning with Residual Weighting Correction

• URL: http://arxiv.org/abs/2606.27112v1
• Date: Thu, 25 Jun 2026 14:48:58 GMT
• Title: Heavy-Ball Q-Learning with Residual Weighting Correction
• Abstract summary: This paper proposes a corrected heavy-ball Q-learning method for reinforcement learning (RL)<n>It also identifies conditions under which the method is theoretically guaranteed to converge faster than standard Q-learning.
• Score: 7.8232617281369805
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This paper proposes a corrected heavy-ball Q-learning method for reinforcement learning (RL) and establishes its convergence. It also identifies conditions under which the method is theoretically guaranteed to converge faster than standard Q-learning. The same construction is then extended to Q-learning with linear function approximation, where analogous convergence and acceleration statements are derived. The analysis is based on a switched linear system (SLS) representation of Q-learning algorithms and on the joint spectral radius (JSR) of the associated switching families. This SLS viewpoint is not commonly used in standard analyses of Q-learning, and it provides a complementary framework and new insight into how heavy-ball momentum can accelerate Q-learning.

Related papers

• Spectral Analysis of Dueling Q-Learning [7.8232617281369805]
  Q-learning is a fundamental algorithm in reinforcement learning (RL) for solving discounted Markov decision processes (MDPs)<n>The deep Q-network (DQN) extends Q-learning by using a deep neural network for Q-function approximation.<n>Dueling Q-learning decomposes the Q-function into a value function and an advantage function and learns the two components jointly.
  arXiv  Detail & Related papers  (2026-07-09T10:29:22Z)
• Target Updates May Stabilize Linear Q-Learning: Periodic and Soft Dynamics [7.8232617281369805]
  This paper gives a rigorous and exact analysis of mechanisms for Q-learning with linear function approximation (linear Q-learning)<n>Although linear Q-learning can fail to converge in general, we prove that, under explicit spectral and step-size conditions, periodic hard target updates and soft target updates can guarantee convergence to the exact projected Q-Bellman solution.
  arXiv  Detail & Related papers  (2026-05-31T15:46:20Z)
• GQA-μP: The maximal parameterization update for grouped query attention [64.99972873396649]
  We show how to derivation P scalings for grouped-query attention (GQA)<n>We also demonstrate the efficacy of our theoretical derivations by showing learning rate transfer across the GQA repetition hyperparameter.
  arXiv  Detail & Related papers  (2026-05-14T18:03:16Z)
• A Switching System Theory of Q-Learning with Linear Function Approximation [10.48957127047368]
  This paper develops a switching-system interpretation of Q-learning with linear function approximation (LFA) based on the joint spectral radius (JSR)<n>We derive an exact linear switched model for the mean dynamics and relate it to the corresponding switched system.<n>The framework also yields a JSR-based view of regularized Q-learning with LFA.
  arXiv  Detail & Related papers  (2026-05-10T16:21:31Z)
• VQC-MLPNet: An Unconventional Hybrid Quantum-Classical Architecture for Scalable and Robust Quantum Machine Learning [50.95799256262098]
  Variational quantum circuits (VQCs) hold promise for quantum machine learning but face challenges in expressivity, trainability, and noise resilience.<n>We propose VQC-MLPNet, a hybrid architecture where a VQC generates the first-layer weights of a classical multilayer perceptron during training, while inference is performed entirely classically.
  arXiv  Detail & Related papers  (2025-06-12T01:38:15Z)
• Unified ODE Analysis of Smooth Q-Learning Algorithms [7.8232617281369805]
  Recently, an convergence analysis for Q-learning was introduced using a switching system framework.<n>We present a more general and unified convergence analysis that improves upon the switching system approach.
  arXiv  Detail & Related papers  (2024-04-20T01:16:27Z)
• Unifying (Quantum) Statistical and Parametrized (Quantum) Algorithms [65.268245109828]
  We take inspiration from Kearns' SQ oracle and Valiant's weak evaluation oracle. We introduce an extensive yet intuitive framework that yields unconditional lower bounds for learning from evaluation queries.
  arXiv  Detail & Related papers  (2023-10-26T18:23:21Z)
• Pre-training Tensor-Train Networks Facilitates Machine Learning with Variational Quantum Circuits [70.97518416003358]
  Variational quantum circuits (VQCs) hold promise for quantum machine learning on noisy intermediate-scale quantum (NISQ) devices. While tensor-train networks (TTNs) can enhance VQC representation and generalization, the resulting hybrid model, TTN-VQC, faces optimization challenges due to the Polyak-Lojasiewicz (PL) condition. To mitigate this challenge, we introduce Pre+TTN-VQC, a pre-trained TTN model combined with a VQC.
  arXiv  Detail & Related papers  (2023-05-18T03:08:18Z)
• Online Target Q-learning with Reverse Experience Replay: Efficiently finding the Optimal Policy for Linear MDPs [50.75812033462294]
  We bridge the gap between practical success of Q-learning and pessimistic theoretical results. We present novel methods Q-Rex and Q-RexDaRe. We show that Q-Rex efficiently finds the optimal policy for linear MDPs.
  arXiv  Detail & Related papers  (2021-10-16T01:47:41Z)
• Finite-Time Analysis of Asynchronous Q-Learning with Discrete-Time Switching System Models [6.85316573653194]
  We prove that Q-learning with a constant step-size can be naturally formulated as discrete-time switched linear systems. It offers novel and intuitive insights on Q-learning mainly based on controloretic frameworks.
  arXiv  Detail & Related papers  (2021-02-17T05:32:07Z)
• Momentum Q-learning with Finite-Sample Convergence Guarantee [49.38471009162477]
  This paper analyzes a class of momentum-based Q-learning algorithms with finite-sample guarantee. We establish the convergence guarantee for MomentumQ with linear function approximations and Markovian sampling. We demonstrate through various experiments that the proposed MomentumQ outperforms other momentum-based Q-learning algorithms.
  arXiv  Detail & Related papers  (2020-07-30T12:27:03Z)
• Analysis of Q-learning with Adaptation and Momentum Restart for Gradient Descent [47.3692506462581]
  We first characterize the convergence rate for Q-AMSGrad, which is the Q-learning algorithm with AMSGrad update. To further improve the performance, we propose to incorporate the momentum restart scheme to Q-AMSGrad, resulting in the so-called Q-AMSGradR algorithm. Our experiments on a linear quadratic regulator problem show that the two proposed Q-learning algorithms outperform the vanilla Q-learning with SGD updates.
  arXiv  Detail & Related papers  (2020-07-15T01:11:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.