Deep Q-Learning on Hölder Spaces

• URL: http://arxiv.org/abs/2606.16846v1
• Date: Mon, 15 Jun 2026 15:23:22 GMT
• Title: Deep Q-Learning on Hölder Spaces
• Abstract summary: We study the operator-theoretic core of Q-learning in continuous-time control with continuous states and actions.<n>We show that a Bellman update maps bounded inputs into an anisotropic regularity class, smoothing the state variable while leaving only Lipschitz dependence on the action variable.<n>The resulting theory makes a direct contribution to Q-learning theory at the level of Bellman target regularity and approximation in continuous control.
• Score: 0.5671339754416556
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We study the operator-theoretic core of Q-learning in continuous-time stochastic control with continuous states and actions. In value-based reinforcement learning, each Q-learning or DQN update is built from a Bellman optimality target; our analysis isolates this target in a diffusion setting and studies its regularity and approximation complexity. Under uniform ellipticity and Hölder-regular coefficients, we show that a Bellman update maps bounded inputs into an anisotropic regularity class, smoothing the state variable while leaving only Lipschitz dependence on the action variable. This yields a compact family of Bellman iterates and motivates a tensor-product DeepONet architecture adapted to the mixed regularity of the problem. We then derive explicit approximation and resource bounds, together with a stiffness--complexity trade-off as the time step $δ\to 0$. The resulting theory makes a direct contribution to Q-learning theory at the level of Bellman target regularity and approximation in continuous stochastic control. At the same time, we do not claim a full convergence theorem for practical sampled Q-learning with exploration, replay, and stochastic gradient updates.

Related papers

• Learning the Supports for Categorical Critic in Reinforcement Learning [44.603295025494674]
  We propose an approach that learns the lower and upper bounds of the support instead of assigning them beforehand.<n>We show that this objective forms an upper bound on the mean-squared Bellman error.<n>Our theoretical analysis shows that this bound is tighter than that of non-learned supports of HL-Gauss.
  arXiv  Detail & Related papers  (2026-07-02T08:34:42Z)
• Global Convergence of Wasserstein Policy Gradient for Entropy-Regularized Reinforcement Learning [7.683866416886673]
  Wasserstein policy gradient (WPG) is a policy optimization method for reinforcement learning.<n>We develop a global convergence theory for WPG by exploiting the Bellman structure of entropy-regularized RL.<n>Our analysis shows that although entropy-regularized RL is not convex in the usual flat sense, the Bellman recursion induces a favorable Polyak--Lojasiewicz-type geometry.
  arXiv  Detail & Related papers  (2026-05-25T17:42:59Z)
• Continual Quantum Architecture Search with Tensor-Train Encoding: Theory and Applications to Signal Processing [68.35481158940401]
  CL-QAS is a continual quantum architecture search framework.<n>It mitigates challenges of costly encoding amplitude and forgetting in variational quantum circuits.<n>It achieves controllable robustness expressivity, sample-efficient generalization, and smooth convergence without barren plateaus.
  arXiv  Detail & Related papers  (2026-01-10T02:36:03Z)
• Bellman Calibration for V-Learning in Offline Reinforcement Learning [40.322273308230606]
  We introduce Iterated Bellman, a simple, model-agnostic, post-hoc procedure for calibrating off-policy value predictions.<n>We adapt classical histogram and isotonic calibration to the dynamic, counterfactual setting.<n>This yields a one-dimensional fitted value scheme that can be applied to any value estimator.
  arXiv  Detail & Related papers  (2025-12-29T18:52:18Z)
• GB-DQN: Gradient Boosted DQN Models for Non-stationary Reinforcement Learning [0.0]
  We propose emphGradient-Boosted Deep Q-Networks (GB-DQN), an adaptive ensemble method that addresses model drift through incremental residual learning.<n>Instead of retraining a single Q-network, GB-DQN constructs an additive ensemble in which each new learner is trained to approximate the Bellman residual of the current ensemble after drift.
  arXiv  Detail & Related papers  (2025-12-18T19:53:50Z)
• Quantum Boltzmann Machines for Sample-Efficient Reinforcement Learning [0.0]
  We introduce theoretically grounded Continuous Semi-Quantum Boltzmann Machines that support continuous-action reinforcement learning.<n>By combining exponential-family priors over visible units with Boltzmann distributions over hidden units, CSQBMs yield a hybrid quantum-classical model that reduces qubit requirements while retaining strong algorithms.
  arXiv  Detail & Related papers  (2025-11-06T22:40:18Z)
• Contraction, Criticality, and Capacity: A Dynamical-Systems Perspective on Echo-State Networks [13.857230672081489]
  We present a unified, dynamical-systems treatment that weaves together functional analysis, random attractor theory and recent neuroscientific findings.<n>First, we prove that the Echo-State Property (wash-out of initial conditions) together with global Lipschitz dynamics necessarily yields the Fading-Memory Property.<n>Second, employing a Stone-Weierstrass strategy we give a streamlined proof that ESNs with nonlinear reservoirs and linear read-outs are dense in the Banach space of causal, time-in fading-memory filters.<n>Third, we quantify computational resources via memory-capacity spectrum, show how
  arXiv  Detail & Related papers  (2025-07-24T14:41:18Z)
• Universal Approximation Theorem for Deep Q-Learning via FBSDE System [2.1756081703276]
  This paper establishes a Universal Approximation Theorem for a class of Deep Q-Networks (DQNs)<n>We show that layers of a deep residual network, conceived as neural operators acting on function spaces, can approximate the action of the Bellman operator.
  arXiv  Detail & Related papers  (2025-05-09T13:11:55Z)
• Provable Guarantees for Generative Behavior Cloning: Bridging Low-Level Stability and High-Level Behavior [51.60683890503293]
  We propose a theoretical framework for studying behavior cloning of complex expert demonstrations using generative modeling. We show that pure supervised cloning can generate trajectories matching the per-time step distribution of arbitrary expert trajectories.
  arXiv  Detail & Related papers  (2023-07-27T04:27:26Z)
• Wasserstein Actor-Critic: Directed Exploration via Optimism for Continuous-Actions Control [41.7453231409493]
  Wasserstein Actor-Critic ( WAC) is an actor-critic architecture inspired by the Wasserstein Q-Learning (WQL) citepwql. WAC enforces exploration in a principled way by guiding the policy learning process with the optimization of an upper bound of the Q-value estimates.
  arXiv  Detail & Related papers  (2023-03-04T10:52:20Z)
• Temporal Difference Learning with Compressed Updates: Error-Feedback meets Reinforcement Learning [47.904127007515925]
  We study a variant of the classical temporal difference (TD) learning algorithm with a perturbed update direction. We prove that compressed TD algorithms, coupled with an error-feedback mechanism used widely in optimization, exhibit the same non-asymptotic approximation guarantees as their counterparts. Notably, these are the first finite-time results in RL that account for general compression operators and error-feedback in tandem with linear function approximation and Markovian sampling.
  arXiv  Detail & Related papers  (2023-01-03T04:09:38Z)
• Tightening the Dependence on Horizon in the Sample Complexity of Q-Learning [59.71676469100807]
  This work sharpens the sample complexity of synchronous Q-learning to an order of $frac|mathcalS|| (1-gamma)4varepsilon2$ for any $0varepsilon 1$. Our finding unveils the effectiveness of vanilla Q-learning, which matches that of speedy Q-learning without requiring extra computation and storage.
  arXiv  Detail & Related papers  (2021-02-12T14:22:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.