A Switching System Theory of Q-Learning with Linear Function Approximation

• URL: http://arxiv.org/abs/2605.11021v1
• Date: Sun, 10 May 2026 16:21:31 GMT
• Title: A Switching System Theory of Q-Learning with Linear Function Approximation
• Abstract summary: This paper develops a switching-system interpretation of Q-learning with linear function approximation (LFA) based on the joint spectral radius (JSR)<n>We derive an exact linear switched model for the mean dynamics and relate it to the corresponding switched system.<n>The framework also yields a JSR-based view of regularized Q-learning with LFA.
• Score: 10.48957127047368
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper develops a switching-system interpretation of Q-learning with linear function approximation (LFA) based on the joint spectral radius (JSR). We derive an exact linear switched model for the mean dynamics and relate convergence to stability of the corresponding switched system. The same construction is then used for stochastic linear Q-learning with independent and identically distributed (i.i.d.) observations and with Markovian observations. Although exact JSR computation is difficult in general, the certificate captures products of switching modes and can be less conservative than one-step norm bounds. The framework also yields a JSR-based view of regularized Q-learning with LFA. The resulting analysis connects projected Bellman equations, finite-difference stochastic-policy switching, and switched-system stability in a single parameter-space formulation.

Related papers

• Equation Recast for Canonical Operator Learning Across Parametric PDEs [59.44573033543284]
  We introduce equation recast, which reformulates parametric operator learning as the learning of a single canonical operator.<n>In high-fidelity tokamak simulations for nuclear fusion, the framework unifies electron-temperature data across four device geometries.
  arXiv  Detail & Related papers  (2026-09-02T13:15:50Z)
• Heavy-Ball Q-Learning with Residual Weighting Correction [7.8232617281369805]
  This paper proposes a corrected heavy-ball Q-learning method for reinforcement learning (RL)<n>It also identifies conditions under which the method is theoretically guaranteed to converge faster than standard Q-learning.
  arXiv  Detail & Related papers  (2026-06-25T14:48:58Z)
• Target Updates May Stabilize Linear Q-Learning: Periodic and Soft Dynamics [7.8232617281369805]
  This paper gives a rigorous and exact analysis of mechanisms for Q-learning with linear function approximation (linear Q-learning)<n>Although linear Q-learning can fail to converge in general, we prove that, under explicit spectral and step-size conditions, periodic hard target updates and soft target updates can guarantee convergence to the exact projected Q-Bellman solution.
  arXiv  Detail & Related papers  (2026-05-31T15:46:20Z)
• Lyapunov-Certified Direct Switching Theory for Q-Learning [7.8232617281369805]
  We analyze Q-learning through a direct switching system representation.<n>We derive a finite-time final-iterate bound via a JSR-induced Lyapunov function.
  arXiv  Detail & Related papers  (2026-04-21T15:22:42Z)
• Disordered Dynamics in High Dimensions: Connections to Random Matrices and Machine Learning [52.26396748560348]
  We provide an overview of high dimensional dynamical systems driven by random matrices.<n>We focus on applications to simple models of learning and generalization in machine learning theory.
  arXiv  Detail & Related papers  (2026-01-03T00:12:32Z)
• Unified ODE Analysis of Smooth Q-Learning Algorithms [7.8232617281369805]
  Recently, an convergence analysis for Q-learning was introduced using a switching system framework.<n>We present a more general and unified convergence analysis that improves upon the switching system approach.
  arXiv  Detail & Related papers  (2024-04-20T01:16:27Z)
• Machine learning in and out of equilibrium [58.88325379746631]
  Our study uses a Fokker-Planck approach, adapted from statistical physics, to explore these parallels. We focus in particular on the stationary state of the system in the long-time limit, which in conventional SGD is out of equilibrium. We propose a new variation of Langevin dynamics (SGLD) that harnesses without replacement minibatching.
  arXiv  Detail & Related papers  (2023-06-06T09:12:49Z)
• Formal Controller Synthesis for Markov Jump Linear Systems with Uncertain Dynamics [64.72260320446158]
  We propose a method for synthesising controllers for Markov jump linear systems. Our method is based on a finite-state abstraction that captures both the discrete (mode-jumping) and continuous (stochastic linear) behaviour of the MJLS. We apply our method to multiple realistic benchmark problems, in particular, a temperature control and an aerial vehicle delivery problem.
  arXiv  Detail & Related papers  (2022-12-01T17:36:30Z)
• Identifiability and Asymptotics in Learning Homogeneous Linear ODE Systems from Discrete Observations [114.17826109037048]
  Ordinary Differential Equations (ODEs) have recently gained a lot of attention in machine learning. theoretical aspects, e.g., identifiability and properties of statistical estimation are still obscure. This paper derives a sufficient condition for the identifiability of homogeneous linear ODE systems from a sequence of equally-spaced error-free observations sampled from a single trajectory.
  arXiv  Detail & Related papers  (2022-10-12T06:46:38Z)
• Exact solutions of interacting dissipative systems via weak symmetries [77.34726150561087]
  We analytically diagonalize the Liouvillian of a class Markovian dissipative systems with arbitrary strong interactions or nonlinearity. This enables an exact description of the full dynamics and dissipative spectrum. Our method is applicable to a variety of other systems, and could provide a powerful new tool for the study of complex driven-dissipative quantum systems.
  arXiv  Detail & Related papers  (2021-09-27T17:45:42Z)
• Finite-Time Analysis of Asynchronous Q-Learning with Discrete-Time Switching System Models [6.85316573653194]
  We prove that Q-learning with a constant step-size can be naturally formulated as discrete-time switched linear systems. It offers novel and intuitive insights on Q-learning mainly based on controloretic frameworks.
  arXiv  Detail & Related papers  (2021-02-17T05:32:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.