A Spectral Revisit of the Distributional Bellman Operator under the Cramér Metric

• URL: http://arxiv.org/abs/2603.12576v1
• Date: Fri, 13 Mar 2026 02:21:54 GMT
• Title: A Spectral Revisit of the Distributional Bellman Operator under the Cramér Metric
• Abstract summary: Distributional reinforcement learning (DRL) studies the evolution of full return distributions under Bellman updates.<n>A classical result is that the distributional Bellman operator is contractive under the Cramér metric.<n>We analyse distributional Bellman dynamics directly at the level of CDFs, treating the Cramér geometry as the intrinsic analytical setting.
• Score: 19.762345326138536
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Distributional reinforcement learning (DRL) studies the evolution of full return distributions under Bellman updates rather than focusing on expected values. A classical result is that the distributional Bellman operator is contractive under the Cramér metric, which corresponds to an $L^2$ geometry on differences of cumulative distribution functions (CDFs). While this contraction ensures stability of policy evaluation, existing analyses remain largely metric, focusing on contraction properties without elucidating the structural action of the Bellman update on distributions. In this work, we analyse distributional Bellman dynamics directly at the level of CDFs, treating the Cramér geometry as the intrinsic analytical setting. At this level, the Bellman update acts affinely on CDFs and linearly on differences between CDFs, and its contraction property yields a uniform bound on this linear action. Building on this intrinsic formulation, we construct a family of regularised spectral Hilbert representations that realise the CDF-level geometry by exact conjugation, without modifying the underlying Bellman dynamics. The regularisation affects only the geometry and vanishes in the zero-regularisation limit, recovering the native Cramér metric. This framework clarifies the operator structure underlying distributional Bellman updates and provides a foundation for further functional and operator-theoretic analyses in DRL.

Related papers

• Distributional Soft Bellman Operator under the Cramér Geometry [19.762345326138536]
  This paper studies the Cramér-geometric Bellman fixed point associated with the policy-evaluation step of DSPI.<n>It provides a reference point for studying approximate critics, evaluation error, and critic-loss design in DSPI-style algorithms.
  arXiv  Detail & Related papers  (2026-07-20T12:44:30Z)
• Deep Q-Learning on Hölder Spaces [0.5671339754416556]
  We study the operator-theoretic core of Q-learning in continuous-time control with continuous states and actions.<n>We show that a Bellman update maps bounded inputs into an anisotropic regularity class, smoothing the state variable while leaving only Lipschitz dependence on the action variable.<n>The resulting theory makes a direct contribution to Q-learning theory at the level of Bellman target regularity and approximation in continuous control.
  arXiv  Detail & Related papers  (2026-06-15T15:23:22Z)
• Global Convergence of Wasserstein Policy Gradient for Entropy-Regularized Reinforcement Learning [7.683866416886673]
  Wasserstein policy gradient (WPG) is a policy optimization method for reinforcement learning.<n>We develop a global convergence theory for WPG by exploiting the Bellman structure of entropy-regularized RL.<n>Our analysis shows that although entropy-regularized RL is not convex in the usual flat sense, the Bellman recursion induces a favorable Polyak--Lojasiewicz-type geometry.
  arXiv  Detail & Related papers  (2026-05-25T17:42:59Z)
• SENTINEL: Stagewise Integrity Verification for Pipeline Parallel Decentralized Training [54.8494905524997]
  Decentralized training introduces critical security risks when executed across untrusted, geographically distributed nodes.<n>We propose SENTINEL, a verification mechanism for pipeline parallelism (PP) training without duplication.<n>Experiments demonstrate successful training of up to 4B- parameter LLMs across untrusted distributed environments with up to 176 workers while maintaining model convergence and performance.
  arXiv  Detail & Related papers  (2026-03-03T23:51:10Z)
• Geometric Convergence Analysis of Variational Inference via Bregman Divergences [3.7098038388802252]
  Vari rigorous Inference (VI) provides a scalable framework for inference by the Lower Evidence (ELBO)<n>We establish a novel theoretical framework for analyzing objective convergence by exploiting the exponential family distributions.
  arXiv  Detail & Related papers  (2025-10-17T11:30:05Z)
• Spectral Bellman Method: Unifying Representation and Exploration in RL [54.71169912483302]
  This work introduces Spectral Bellman Representation, a novel framework for learning representations for value-based reinforcement learning.<n>We show that our learned representations enable structured exploration, by aligning feature covariance with Bellman dynamics.<n>Our framework naturally extends to powerful multi-step Bellman operators, further broadening its impact.
  arXiv  Detail & Related papers  (2025-07-17T14:50:52Z)
• Universal Approximation Theorem for Deep Q-Learning via FBSDE System [2.1756081703276]
  This paper establishes a Universal Approximation Theorem for a class of Deep Q-Networks (DQNs)<n>We show that layers of a deep residual network, conceived as neural operators acting on function spaces, can approximate the action of the Bellman operator.
  arXiv  Detail & Related papers  (2025-05-09T13:11:55Z)
• Bellman Diffusion: Generative Modeling as Learning a Linear Operator in the Distribution Space [72.52365911990935]
  We introduce Bellman Diffusion, a novel DGM framework that maintains linearity in MDPs through gradient and scalar field modeling. Our results show that Bellman Diffusion achieves accurate field estimations and is a capable image generator, converging 1.5x faster than the traditional histogram-based baseline in distributional RL tasks.
  arXiv  Detail & Related papers  (2024-10-02T17:53:23Z)
• Value-Distributional Model-Based Reinforcement Learning [59.758009422067]
  Quantifying uncertainty about a policy's long-term performance is important to solve sequential decision-making tasks. We study the problem from a model-based Bayesian reinforcement learning perspective. We propose Epistemic Quantile-Regression (EQR), a model-based algorithm that learns a value distribution function.
  arXiv  Detail & Related papers  (2023-08-12T14:59:19Z)
• LLQL: Logistic Likelihood Q-Learning for Reinforcement Learning [1.5734309088976395]
  This study investigates the distribution of the Bellman approximation error through iterative exploration of the Bellman equation. We propose the utilization of the Logistic maximum likelihood function (LLoss) as an alternative to the commonly used mean squared error (MSELoss) that assumes a Normal distribution for Bellman errors.
  arXiv  Detail & Related papers  (2023-07-05T15:00:29Z)
• Optimal policy evaluation using kernel-based temporal difference methods [78.83926562536791]
  We use kernel Hilbert spaces for estimating the value function of an infinite-horizon discounted Markov reward process. We derive a non-asymptotic upper bound on the error with explicit dependence on the eigenvalues of the associated kernel operator. We prove minimax lower bounds over sub-classes of MRPs.
  arXiv  Detail & Related papers  (2021-09-24T14:48:20Z)
• Bayesian Bellman Operators [55.959376449737405]
  We introduce a novel perspective on Bayesian reinforcement learning (RL) Our framework is motivated by the insight that when bootstrapping is introduced, model-free approaches actually infer a posterior over Bellman operators, not value functions.
  arXiv  Detail & Related papers  (2021-06-09T12:20:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.