Generalised Bellman recurrence and three dualities in sequential decision-making

• URL: http://arxiv.org/abs/2607.18077v1
• Date: Mon, 20 Jul 2026 15:46:19 GMT
• Title: Generalised Bellman recurrence and three dualities in sequential decision-making
• Abstract summary: We show that the Bellman equation arises from three conditions.<n>The same conditions give rise to three dualities: one between probability and return, one between return and aggregation, and one between aggregation and probability.<n>Our framework reveals these dualities as arising from a single construction, unifying methods developed separately across reinforcement learning, control, and decision theory.
• Score: 40.8583449341828
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: What gives the Bellman equation its form? We show that the recursive properties of optimal value functions follow from three conditions: that the dynamics decomposes through sufficient statistics, that the return decomposes recursively, and that the aggregation of uncertainty is compatible with both. When all three conditions hold on a common state, the Bellman equation arises from their mutual consistency; when one fails, tractability can often be recovered by augmenting the state or by deforming return or dynamics. The same conditions are shown to give rise to three dualities: one between probability and return, one between return and aggregation, and one between aggregation and probability. Our framework reveals these dualities as arising from a single construction, unifying methods developed separately across reinforcement learning, control, and decision theory.

Related papers

• Curated Synthetic Data Doesn't Have to Collapse: A Theoretical Study of Generative Retraining with Pluralistic Preferences [13.835275211048113]
  Recursive retraining of generative models poses a critical representation challenge.<n>We show that collapse can be mitigated through curation based on multiple reward functions.
  arXiv  Detail & Related papers  (2026-05-08T13:27:23Z)
• On the Mechanism and Dynamics of Modular Addition: Fourier Features, Lottery Ticket, and Grokking [49.1352577985191]
  We present a comprehensive analysis of how two-layer neural networks learn features to solve the modular addition task.<n>Our work provides a full mechanistic interpretation of the learned model and a theoretical explanation of its training dynamics.
  arXiv  Detail & Related papers  (2026-02-18T20:25:13Z)
• The Alignment Game: A Theory of Long-Horizon Alignment Through Recursive Curation [13.835275211048113]
  We model alignment as an interaction between two factions: the Model Owner, who filters which outputs should be learned by the model, and the Public User, who determines which outputs are ultimately shared and retained through interactions with the model.<n>Our analysis reveals three structural convergence regimes depending on the degree of preference alignment: consensus collapse, compromise on shared optima, and asymmetric refinement.<n>We show that alignment is not a static goal but an evolving equilibrium, shaped both by power asymmetries and path dependence.
  arXiv  Detail & Related papers  (2025-11-16T22:17:16Z)
• An open harmonic chain: Exact vs global and local reduced dynamics [0.0]
  We derive the steady states of the open chain quantum dynamics in two approaches.<n>We show the existence of a temperature-dependent critical inter-oscillator coupling strength that determines the domain of validity of each approach.
  arXiv  Detail & Related papers  (2025-05-22T13:33:00Z)
• Understanding and Constructing Latent Modality Structures in Multi-modal Representation Learning [53.68371566336254]
  We argue that the key to better performance lies in meaningful latent modality structures instead of perfect modality alignment. Specifically, we design 1) a deep feature separation loss for intra-modality regularization; 2) a Brownian-bridge loss for inter-modality regularization; and 3) a geometric consistency loss for both intra- and inter-modality regularization.
  arXiv  Detail & Related papers  (2023-03-10T14:38:49Z)
• On the Pointwise Behavior of Recursive Partitioning and Its Implications for Heterogeneous Causal Effect Estimation [8.394633341978007]
  Decision tree learning is increasingly being used for pointwise inference. We show that adaptive decision trees can fail to achieve convergence rates of convergence in the norm with non-vanishing probability. We show that random forests can remedy the situation, turning poor performing trees into nearly optimal procedures.
  arXiv  Detail & Related papers  (2022-11-19T21:28:30Z)
• Growth of entanglement of generic states under dual-unitary dynamics [77.34726150561087]
  Dual-unitary circuits are a class of locally-interacting quantum many-body systems. In particular, they admit a class of solvable" initial states for which, in the thermodynamic limit, one can access the full non-equilibrium dynamics. We show that in this case the entanglement increment during a time step is sub-maximal for finite times, however, it approaches the maximal value in the infinite-time limit.
  arXiv  Detail & Related papers  (2022-07-29T18:20:09Z)
• Action Redundancy in Reinforcement Learning [54.291331971813364]
  We show that transition entropy can be described by two terms; namely, model-dependent transition entropy and action redundancy. Our results suggest that action redundancy is a fundamental problem in reinforcement learning.
  arXiv  Detail & Related papers  (2021-02-22T19:47:26Z)
• Continuity of Generalized Entropy and Statistical Learning [4.568777157687959]
  We study the continuity property of the generalized entropy as a functional of the underlying probability distribution. We use this property to answer the basic questions in statistical learning theory, the excess risk analyses for various learning methods.
  arXiv  Detail & Related papers  (2020-12-31T18:51:45Z)
• A Critical View of the Structural Causal Model [89.43277111586258]
  We show that one can identify the cause and the effect without considering their interaction at all. We propose a new adversarial training method that mimics the disentangled structure of the causal model. Our multidimensional method outperforms the literature methods on both synthetic and real world datasets.
  arXiv  Detail & Related papers  (2020-02-23T22:52:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.