Spectral Bellman Method: Unifying Representation and Exploration in RL

• URL: http://arxiv.org/abs/2507.13181v1
• Date: Thu, 17 Jul 2025 14:50:52 GMT
• Title: Spectral Bellman Method: Unifying Representation and Exploration in RL
• Authors: Ofir Nabati, Bo Dai, Shie Mannor, Guy Tennenholtz,
• Abstract summary: This work introduces Spectral Bellman Representation, a novel framework for learning representations for value-based reinforcement learning.<n>We show that our learned representations enable structured exploration, by aligning feature covariance with Bellman dynamics.<n>Our framework naturally extends to powerful multi-step Bellman operators, further broadening its impact.
• Score: 54.71169912483302
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The effect of representation has been demonstrated in reinforcement learning, from both theoretical and empirical successes. However, the existing representation learning mainly induced from model learning aspects, misaligning with our RL tasks. This work introduces Spectral Bellman Representation, a novel framework derived from the Inherent Bellman Error (IBE) condition, which aligns with the fundamental structure of Bellman updates across a space of possible value functions, therefore, directly towards value-based RL. Our key insight is the discovery of a fundamental spectral relationship: under the zero-IBE condition, the transformation of a distribution of value functions by the Bellman operator is intrinsically linked to the feature covariance structure. This spectral connection yields a new, theoretically-grounded objective for learning state-action features that inherently capture this Bellman-aligned covariance. Our method requires a simple modification to existing algorithms. We demonstrate that our learned representations enable structured exploration, by aligning feature covariance with Bellman dynamics, and improve overall performance, particularly in challenging hard-exploration and long-horizon credit assignment tasks. Our framework naturally extends to powerful multi-step Bellman operators, further broadening its impact. Spectral Bellman Representation offers a principled and effective path toward learning more powerful and structurally sound representations for value-based reinforcement learning.

Related papers

• AURORA: Augmented Understanding via Structured Reasoning and Reinforcement Learning for Reference Audio-Visual Segmentation [113.75682363364004]
  AURORA is a framework designed to enhance genuine reasoning and language comprehension in reference audio-visual segmentation.<n>AURORA achieves state-of-the-art performance on Ref-AVS benchmarks and generalizes effectively to unreferenced segmentation.
  arXiv  Detail & Related papers  (2025-08-04T07:47:38Z)
• Gradual Transition from Bellman Optimality Operator to Bellman Operator in Online Reinforcement Learning [47.57615889991631]
  For continuous action spaces, actor-critic methods are widely used in online reinforcement learning (RL)<n>This study examines the effectiveness of incorporating the Bellman optimality operator into actor-critic frameworks.
  arXiv  Detail & Related papers  (2025-06-06T10:46:20Z)
• Universal Approximation Theorem for Deep Q-Learning via FBSDE System [2.1756081703276]
  This paper establishes a Universal Approximation Theorem for a class of Deep Q-Networks (DQNs)<n>We show that layers of a deep residual network, conceived as neural operators acting on function spaces, can approximate the action of the Bellman operator.
  arXiv  Detail & Related papers  (2025-05-09T13:11:55Z)
• When is Realizability Sufficient for Off-Policy Reinforcement Learning? [17.317841035807696]
  We analyze the statistical complexity of off-policy reinforcement learning when only realizability holds for the prescribed function class. We establish finite-sample guarantees for off-policy reinforcement learning that are free of the approximation error term known as inherent Bellman error.
  arXiv  Detail & Related papers  (2022-11-10T03:15:31Z)
• Spectral Decomposition Representation for Reinforcement Learning [100.0424588013549]
  We propose an alternative spectral method, Spectral Decomposition Representation (SPEDER), that extracts a state-action abstraction from the dynamics without inducing spurious dependence on the data collection policy. A theoretical analysis establishes the sample efficiency of the proposed algorithm in both the online and offline settings. An experimental investigation demonstrates superior performance over current state-of-the-art algorithms across several benchmarks.
  arXiv  Detail & Related papers  (2022-08-19T19:01:30Z)
• Learning Bellman Complete Representations for Offline Policy Evaluation [51.96704525783913]
  Two sufficient conditions for sample-efficient OPE are Bellman completeness and coverage. We show our representation enables better OPE compared to previous representation learning methods developed for off-policy RL.
  arXiv  Detail & Related papers  (2022-07-12T21:02:02Z)
• A Free Lunch from the Noise: Provable and Practical Exploration for Representation Learning [55.048010996144036]
  We show that under some noise assumption, we can obtain the linear spectral feature of its corresponding Markov transition operator in closed-form for free. We propose Spectral Dynamics Embedding (SPEDE), which breaks the trade-off and completes optimistic exploration for representation learning by exploiting the structure of the noise.
  arXiv  Detail & Related papers  (2021-11-22T19:24:57Z)
• Bayesian Bellman Operators [55.959376449737405]
  We introduce a novel perspective on Bayesian reinforcement learning (RL) Our framework is motivated by the insight that when bootstrapping is introduced, model-free approaches actually infer a posterior over Bellman operators, not value functions.
  arXiv  Detail & Related papers  (2021-06-09T12:20:46Z)
• Neurally Augmented ALISTA [15.021419552695066]
  We introduce Neurally Augmented ALISTA, in which an LSTM network is used to compute step sizes and thresholds individually for each target vector during reconstruction. We show that our approach further improves empirical performance in sparse reconstruction, in particular outperforming existing algorithms by an increasing margin as the compression ratio becomes more challenging.
  arXiv  Detail & Related papers  (2020-10-05T11:39:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.