Bellman Value Decomposition for Task Logic in Safe Optimal Control

• URL: http://arxiv.org/abs/2602.19532v1
• Date: Mon, 23 Feb 2026 05:48:58 GMT
• Title: Bellman Value Decomposition for Task Logic in Safe Optimal Control
• Abstract summary: We consider the innate structure of the Bellman Value as a means to naturally organize the problem for improved automatic performance.<n> Namely, we prove the Bellman Value for a complex task defined in temporal logic can be decomposed into a graph of Bellman Values.<n>To solve the Value and optimal policy, we propose VDPPO, which embeds the decomposed Value graph into a two-layer neural net.
• Score: 18.32953901945506
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Real-world tasks involve nuanced combinations of goal and safety specifications. In high dimensions, the challenge is exacerbated: formal automata become cumbersome, and the combination of sparse rewards tends to require laborious tuning. In this work, we consider the innate structure of the Bellman Value as a means to naturally organize the problem for improved automatic performance. Namely, we prove the Bellman Value for a complex task defined in temporal logic can be decomposed into a graph of Bellman Values, connected by a set of well-known Bellman equations (BEs): the Reach-Avoid BE, the Avoid BE, and a novel type, the Reach-Avoid-Loop BE. To solve the Value and optimal policy, we propose VDPPO, which embeds the decomposed Value graph into a two-layer neural net, bootstrapping the implicit dependencies. We conduct a variety of simulated and hardware experiments to test our method on complex, high-dimensional tasks involving heterogeneous teams and nonlinear dynamics. Ultimately, we find this approach greatly improves performance over existing baselines, balancing safety and liveness automatically.

Related papers

• Deep Q-Learning on Hölder Spaces [0.5671339754416556]
  We study the operator-theoretic core of Q-learning in continuous-time control with continuous states and actions.<n>We show that a Bellman update maps bounded inputs into an anisotropic regularity class, smoothing the state variable while leaving only Lipschitz dependence on the action variable.<n>The resulting theory makes a direct contribution to Q-learning theory at the level of Bellman target regularity and approximation in continuous control.
  arXiv  Detail & Related papers  (2026-06-15T15:23:22Z)
• Latent Spherical Flow Policy for Reinforcement Learning with Combinatorial Actions [31.697208397735395]
  Existing approaches embed task-specific value functions into constrained optimization programs or learn deterministic structured policies, sacrificing generality and policy expressiveness.<n>We propose a solver-induced emphlatent spherical flow policy that brings the expressiveness of modern generative policies to the RL while guaranteeing feasibility by design.<n>Our approach outperforms state-of-the-art baselines by an average of 20.6% across a range of challenging RL tasks.
  arXiv  Detail & Related papers  (2026-01-29T18:49:07Z)
• Networked Restless Multi-Arm Bandits with Reinforcement Learning [4.0539039756740785]
  This paper introduces Networked RMAB, a novel framework that integrates the RMAB model with the independent cascade model.<n>We present its computational challenge due to exponentially large action and state spaces.<n>We experimentally verify these results by developing an efficient Q-learning algorithm tailored to the networked setting.
  arXiv  Detail & Related papers  (2025-12-06T03:53:25Z)
• Spectral Bellman Method: Unifying Representation and Exploration in RL [54.71169912483302]
  This work introduces Spectral Bellman Representation, a novel framework for learning representations for value-based reinforcement learning.<n>We show that our learned representations enable structured exploration, by aligning feature covariance with Bellman dynamics.<n>Our framework naturally extends to powerful multi-step Bellman operators, further broadening its impact.
  arXiv  Detail & Related papers  (2025-07-17T14:50:52Z)
• Entropy-Regularized Token-Level Policy Optimization for Language Agent Reinforcement [67.1393112206885]
  Large Language Models (LLMs) have shown promise as intelligent agents in interactive decision-making tasks. We introduce Entropy-Regularized Token-level Policy Optimization (ETPO), an entropy-augmented RL method tailored for optimizing LLMs at the token level. We assess the effectiveness of ETPO within a simulated environment that models data science code generation as a series of multi-step interactive tasks.
  arXiv  Detail & Related papers  (2024-02-09T07:45:26Z)
• Parameterized Projected Bellman Operator [64.129598593852]
  Approximate value iteration (AVI) is a family of algorithms for reinforcement learning (RL) We propose a novel alternative approach based on learning an approximate version of the Bellman operator. We formulate an optimization problem to learn PBO for generic sequential decision-making problems.
  arXiv  Detail & Related papers  (2023-12-20T09:33:16Z)
• When is Realizability Sufficient for Off-Policy Reinforcement Learning? [17.317841035807696]
  We analyze the statistical complexity of off-policy reinforcement learning when only realizability holds for the prescribed function class. We establish finite-sample guarantees for off-policy reinforcement learning that are free of the approximation error term known as inherent Bellman error.
  arXiv  Detail & Related papers  (2022-11-10T03:15:31Z)
• Log Barriers for Safe Black-box Optimization with Application to Safe Reinforcement Learning [72.97229770329214]
  We introduce a general approach for seeking high dimensional non-linear optimization problems in which maintaining safety during learning is crucial. Our approach called LBSGD is based on applying a logarithmic barrier approximation with a carefully chosen step size. We demonstrate the effectiveness of our approach on minimizing violation in policy tasks in safe reinforcement learning.
  arXiv  Detail & Related papers  (2022-07-21T11:14:47Z)
• Continuous-Time Fitted Value Iteration for Robust Policies [93.25997466553929]
  Solving the Hamilton-Jacobi-Bellman equation is important in many domains including control, robotics and economics. We propose continuous fitted value iteration (cFVI) and robust fitted value iteration (rFVI) These algorithms leverage the non-linear control-affine dynamics and separable state and action reward of many continuous control problems.
  arXiv  Detail & Related papers  (2021-10-05T11:33:37Z)
• Robust Value Iteration for Continuous Control Tasks [99.00362538261972]
  When transferring a control policy from simulation to a physical system, the policy needs to be robust to variations in the dynamics to perform well. We present Robust Fitted Value Iteration, which uses dynamic programming to compute the optimal value function on the compact state domain. We show that robust value is more robust compared to deep reinforcement learning algorithm and the non-robust version of the algorithm.
  arXiv  Detail & Related papers  (2021-05-25T19:48:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.