Target Updates May Stabilize Linear Q-Learning: Periodic and Soft Dynamics

• URL: http://arxiv.org/abs/2606.02645v1
• Date: Sun, 31 May 2026 15:46:20 GMT
• Title: Target Updates May Stabilize Linear Q-Learning: Periodic and Soft Dynamics
• Abstract summary: This paper gives a rigorous and exact analysis of mechanisms for Q-learning with linear function approximation (linear Q-learning)<n>Although linear Q-learning can fail to converge in general, we prove that, under explicit spectral and step-size conditions, periodic hard target updates and soft target updates can guarantee convergence to the exact projected Q-Bellman solution.
• Score: 7.8232617281369805
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Periodic target updates in Q-learning and soft target updates in actor-critic methods are empirically well established stabilization mechanisms, but their precise theoretical explanation is still incomplete. This paper gives a rigorous and exact analysis of these mechanisms for Q-learning with linear function approximation (linear Q-learning) using the exact switched linear system (SLS) dynamics induced by the Bellman maximum and the joint spectral radius (JSR) of the resulting switching matrix families. Although linear Q-learning can fail to converge in general, we prove that, under explicit spectral and step-size conditions, periodic hard target updates and soft target updates can guarantee convergence to the exact projected Q-Bellman solution. The main analysis is carried out for deterministic linear Q-learning, where the target-update mechanism is most transparent. Once the corresponding JSR certificate is established for the mean recursion, the stochastic reinforcement-learning setting can be treated by replacing deterministic modes with sampled stochastic modes and adding the corresponding stochastic-noise analysis.

Related papers

• Heavy-Ball Q-Learning with Residual Weighting Correction [7.8232617281369805]
  This paper proposes a corrected heavy-ball Q-learning method for reinforcement learning (RL)<n>It also identifies conditions under which the method is theoretically guaranteed to converge faster than standard Q-learning.
  arXiv  Detail & Related papers  (2026-06-25T14:48:58Z)
• Geometrically Averaged Hard Target Updates for Linear Q-Learning [7.8232617281369805]
  Recent studies suggest that target updates can improve stability in Q-learning with function approximation, including linear function approximation.<n>We introduce and analyze the so-called $$-target update, obtained by averaging the $m$-periodic target update maps with $$-geometric weights.
  arXiv  Detail & Related papers  (2026-06-09T13:24:36Z)
• A Switching System Theory of Q-Learning with Linear Function Approximation [10.48957127047368]
  This paper develops a switching-system interpretation of Q-learning with linear function approximation (LFA) based on the joint spectral radius (JSR)<n>We derive an exact linear switched model for the mean dynamics and relate it to the corresponding switched system.<n>The framework also yields a JSR-based view of regularized Q-learning with LFA.
  arXiv  Detail & Related papers  (2026-05-10T16:21:31Z)
• Multimodal Learning on Low-Quality Data with Conformal Predictive Self-Calibration [72.0672328514289]
  Multimodal learning often grapples with the challenge of low-quality data.<n>We propose a unified framework, termed Conformal Predictive Self-Calibration.<n>Our framework consistently outperforms existing state-of-the-art methods.
  arXiv  Detail & Related papers  (2026-05-05T14:48:52Z)
• Continual Quantum Architecture Search with Tensor-Train Encoding: Theory and Applications to Signal Processing [68.35481158940401]
  CL-QAS is a continual quantum architecture search framework.<n>It mitigates challenges of costly encoding amplitude and forgetting in variational quantum circuits.<n>It achieves controllable robustness expressivity, sample-efficient generalization, and smooth convergence without barren plateaus.
  arXiv  Detail & Related papers  (2026-01-10T02:36:03Z)
• Disordered Dynamics in High Dimensions: Connections to Random Matrices and Machine Learning [52.26396748560348]
  We provide an overview of high dimensional dynamical systems driven by random matrices.<n>We focus on applications to simple models of learning and generalization in machine learning theory.
  arXiv  Detail & Related papers  (2026-01-03T00:12:32Z)
• Unified ODE Analysis of Smooth Q-Learning Algorithms [7.8232617281369805]
  Recently, an convergence analysis for Q-learning was introduced using a switching system framework.<n>We present a more general and unified convergence analysis that improves upon the switching system approach.
  arXiv  Detail & Related papers  (2024-04-20T01:16:27Z)
• Kalman Filter for Online Classification of Non-Stationary Data [101.26838049872651]
  In Online Continual Learning (OCL) a learning system receives a stream of data and sequentially performs prediction and training steps. We introduce a probabilistic Bayesian online learning model by using a neural representation and a state space model over the linear predictor weights. In experiments in multi-class classification we demonstrate the predictive ability of the model and its flexibility to capture non-stationarity.
  arXiv  Detail & Related papers  (2023-06-14T11:41:42Z)
• Formal Controller Synthesis for Markov Jump Linear Systems with Uncertain Dynamics [64.72260320446158]
  We propose a method for synthesising controllers for Markov jump linear systems. Our method is based on a finite-state abstraction that captures both the discrete (mode-jumping) and continuous (stochastic linear) behaviour of the MJLS. We apply our method to multiple realistic benchmark problems, in particular, a temperature control and an aerial vehicle delivery problem.
  arXiv  Detail & Related papers  (2022-12-01T17:36:30Z)
• Online Target Q-learning with Reverse Experience Replay: Efficiently finding the Optimal Policy for Linear MDPs [50.75812033462294]
  We bridge the gap between practical success of Q-learning and pessimistic theoretical results. We present novel methods Q-Rex and Q-RexDaRe. We show that Q-Rex efficiently finds the optimal policy for linear MDPs.
  arXiv  Detail & Related papers  (2021-10-16T01:47:41Z)
• Finite-Time Analysis of Asynchronous Q-Learning with Discrete-Time Switching System Models [6.85316573653194]
  We prove that Q-learning with a constant step-size can be naturally formulated as discrete-time switched linear systems. It offers novel and intuitive insights on Q-learning mainly based on controloretic frameworks.
  arXiv  Detail & Related papers  (2021-02-17T05:32:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.