Analysis of Control Bellman Residual Minimization for Markov Decision Problem

• URL: http://arxiv.org/abs/2601.18840v1
• Date: Mon, 26 Jan 2026 10:58:27 GMT
• Title: Analysis of Control Bellman Residual Minimization for Markov Decision Problem
• Abstract summary: Bellman residual minimization can be used to minimize the squared Bellman residual objective function.<n>We establish foundational results for the control Bellman residual minimization for policy optimization.
• Score: 6.622208195193136
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Markov decision problems are most commonly solved via dynamic programming. Another approach is Bellman residual minimization, which directly minimizes the squared Bellman residual objective function. However, compared to dynamic programming, this approach has received relatively less attention, mainly because it is often less efficient in practice and can be more difficult to extend to model-free settings such as reinforcement learning. Nonetheless, Bellman residual minimization has several advantages that make it worth investigating, such as more stable convergence with function approximation for value functions. While Bellman residual methods for policy evaluation have been widely studied, methods for policy optimization (control tasks) have been scarcely explored. In this paper, we establish foundational results for the control Bellman residual minimization for policy optimization.

Related papers

• Contraction-Aligned Analysis of Soft Bellman Residual Minimization with Weighted Lp-Norm for Markov Decision Problem [9.333190920811626]
  We consider a soft formulation of Bellman residual minimization and extend it to a generalized weighted Lp -norm.<n>We show that this formulation aligns the optimization objective with the contraction geometry of the Bellman operator as p increases.<n>Our analysis provides a principled connection between residual minimization and Bellman contraction, leading to improved control of error propagation.
  arXiv  Detail & Related papers  (2026-04-08T08:58:30Z)
• Stability and Generalization for Bellman Residuals [8.250374560598493]
  Bellman residual minimization (BRM) has emerged as an attractive remedy for offline reinforcement learning.<n>This paper introduces a single Lyapunov potential that couples SGDA runs on neighbouring datasets and yields an O (1/n) on-average argument-stability bound.<n>The results hold for standard neural-network parameterizations and minibatch SGD.
  arXiv  Detail & Related papers  (2025-08-26T07:15:36Z)
• Gradual Transition from Bellman Optimality Operator to Bellman Operator in Online Reinforcement Learning [47.57615889991631]
  For continuous action spaces, actor-critic methods are widely used in online reinforcement learning (RL)<n>This study examines the effectiveness of incorporating the Bellman optimality operator into actor-critic frameworks.
  arXiv  Detail & Related papers  (2025-06-06T10:46:20Z)
• Trajectory Bellman Residual Minimization: A Simple Value-Based Method for LLM Reasoning [55.33984461046492]
  Policy-based methods currently dominate reinforcement learning pipelines for large language model (LLM) reasoning.<n>We introduce Trajectory Bellman Residual Minimization (TBRM), an algorithm that naturally adapts this idea to LLMs.<n>We prove convergence to the near-optimal KL-regularized policy from arbitrary off-policy via an improved change-of-trajectory-measure analysis.
  arXiv  Detail & Related papers  (2025-05-21T09:41:53Z)
• Value-Distributional Model-Based Reinforcement Learning [59.758009422067]
  Quantifying uncertainty about a policy's long-term performance is important to solve sequential decision-making tasks. We study the problem from a model-based Bayesian reinforcement learning perspective. We propose Epistemic Quantile-Regression (EQR), a model-based algorithm that learns a value distribution function.
  arXiv  Detail & Related papers  (2023-08-12T14:59:19Z)
• When is Realizability Sufficient for Off-Policy Reinforcement Learning? [17.317841035807696]
  We analyze the statistical complexity of off-policy reinforcement learning when only realizability holds for the prescribed function class. We establish finite-sample guarantees for off-policy reinforcement learning that are free of the approximation error term known as inherent Bellman error.
  arXiv  Detail & Related papers  (2022-11-10T03:15:31Z)
• Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learning [85.50033812217254]
  Actor-critic methods are widely used in offline reinforcement learning practice, but are not so well-understood theoretically. We propose a new offline actor-critic algorithm that naturally incorporates the pessimism principle.
  arXiv  Detail & Related papers  (2021-08-19T17:27:29Z)
• A Generalized Projected Bellman Error for Off-policy Value Estimation in Reinforcement Learning [25.39784277231972]
  We introduce a new generalized MSPBE that extends the linear MSPBE to the nonlinear setting. We derive an easy-to-use, but sound, algorithm to minimize the generalized objective.
  arXiv  Detail & Related papers  (2021-04-28T15:50:34Z)
• Logistic Q-Learning [87.00813469969167]
  We propose a new reinforcement learning algorithm derived from a regularized linear-programming formulation of optimal control in MDPs. The main feature of our algorithm is a convex loss function for policy evaluation that serves as a theoretically sound alternative to the widely used squared Bellman error.
  arXiv  Detail & Related papers  (2020-10-21T17:14:31Z)
• Kalman meets Bellman: Improving Policy Evaluation through Value Tracking [59.691919635037216]
  Policy evaluation is a key process in Reinforcement Learning (RL) We devise an optimization method, called Kalman Optimization for Value Approximation (KOVA) KOVA minimizes a regularized objective function that concerns both parameter and noisy return uncertainties.
  arXiv  Detail & Related papers  (2020-02-17T13:30:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.