Maturing Markov Decision Processes: Decision Making under Increasing Information and Shrinking Action Sets

• URL: http://arxiv.org/abs/2606.18820v1
• Date: Wed, 17 Jun 2026 08:41:55 GMT
• Title: Maturing Markov Decision Processes: Decision Making under Increasing Information and Shrinking Action Sets
• Abstract summary: Sequential decision problems often exhibit an asymmetric evolution of information and decision flexibility.<n>We introduce Maturing Markov Decision Processes (MMDPs), a formulation built around this information--action asymmetry.<n>We develop a structure-aware reinforcement learning framework with stage-aware policy design, expiring-action abstraction, and search-augmented learning with distillation.
• Score: 10.02149184877298
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Sequential decision problems often exhibit an asymmetric evolution of information and decision flexibility: as a decision cycle unfolds, the agent receives richer information while feasible actions expire due to operational cutoffs, commitments, or resource constraints. Standard MDP formulations typically flatten this structure into stage-dependent state descriptions and action masks, thereby obscuring the nested information--action asymmetry that determines which decisions are urgent and which can be deferred. We introduce Maturing Markov Decision Processes (MMDPs), a formulation built around this information--action asymmetry. We characterize one of its key consequences through an expiring-action priority principle, which identifies the actions that must be resolved before the next stage. Motivated by this structure, we develop a structure-aware reinforcement learning framework with stage-aware policy design, expiring-action abstraction, and search-augmented learning with distillation. Experiments on a controlled multi-supplier replenishment problem, simplified cash-management environments of increasing complexity, and a production-scale simulator show that explicitly modeling this asymmetry improves learning efficiency and becomes increasingly valuable as decision problems scale.

Related papers

• CLAIM: Leading Open-domain Active Clarification of Large Language Models with Uncertainty Measurement [50.91045324738942]
  CLAIM is an uncertainty-driven framework for active clarification learning in open-domain settings.<n>We propose an entropy-driven synthetic data generation pipeline that integrates uncertainty estimation with semantic clustering and reasoning-based judgments.<n> Experimental results demonstrate that CLAIM can learn stable and generalizable clarification strategies without relying on manually labeled data.
  arXiv  Detail & Related papers  (2026-08-12T04:27:45Z)
• DecisionFlow: Advancing Large Language Model as Principled Decision Maker [49.088778182807395]
  DecisionFlow is a novel decision modeling framework that guides models to reason over structured representations of actions, attributes, and constraints.<n>Rather than predicting answers directly from prompts, DecisionFlow builds a semantically grounded decision space and infers a latent utility function.<n> Empirical results show that DecisionFlow achieves up to 30% accuracy gains over strong prompting baselines.
  arXiv  Detail & Related papers  (2025-05-27T16:23:53Z)
• Revelations: A Decidable Class of POMDPs with Omega-Regular Objectives [5.696882834567791]
  Partially observable Markov decision processes (POMDPs) form a prominent model for uncertainty in sequential decision making.<n>We introduce a revelation mechanism which restricts information loss by requiring that almost surely the agent has eventually full information of the current state.<n>This yields a conceptually simple and exact algorithm for a large class of POMDPs.
  arXiv  Detail & Related papers  (2024-12-16T18:37:06Z)
• Reinforcement Learning with a Terminator [80.34572413850186]
  We learn the parameters of the TerMDP and leverage the structure of the estimation problem to provide state-wise confidence bounds. We use these to construct a provably-efficient algorithm, which accounts for termination, and bound its regret.
  arXiv  Detail & Related papers  (2022-05-30T18:40:28Z)
• Inverse Online Learning: Understanding Non-Stationary and Reactionary Policies [79.60322329952453]
  We show how to develop interpretable representations of how agents make decisions. By understanding the decision-making processes underlying a set of observed trajectories, we cast the policy inference problem as the inverse to this online learning problem. We introduce a practical algorithm for retrospectively estimating such perceived effects, alongside the process through which agents update them. Through application to the analysis of UNOS organ donation acceptance decisions, we demonstrate that our approach can bring valuable insights into the factors that govern decision processes and how they change over time.
  arXiv  Detail & Related papers  (2022-03-14T17:40:42Z)
• Counterfactual Explanations in Sequential Decision Making Under Uncertainty [27.763369810430653]
  We develop methods to find counterfactual explanations for sequential decision making processes. In our problem formulation, the counterfactual explanation specifies an alternative sequence of actions differing in at most k actions. We show that our algorithm finds can provide valuable insights to enhance decision making under uncertainty.
  arXiv  Detail & Related papers  (2021-07-06T17:38:19Z)
• Probabilistic Loss and its Online Characterization for Simplified Decision Making Under Uncertainty [13.807859854345834]
  We extend the decision making mechanism to the whole by removing standard approximations and considering all previously suppressed sources of variability. We present novel bounds on the return and characterize online the effect of simplification using this framework on a particular simplification technique.
  arXiv  Detail & Related papers  (2021-05-12T17:02:01Z)
• Stein Variational Model Predictive Control [130.60527864489168]
  Decision making under uncertainty is critical to real-world, autonomous systems. Model Predictive Control (MPC) methods have demonstrated favorable performance in practice, but remain limited when dealing with complex distributions. We show that this framework leads to successful planning in challenging, non optimal control problems.
  arXiv  Detail & Related papers  (2020-11-15T22:36:59Z)
• Deep reinforcement learning driven inspection and maintenance planning under incomplete information and constraints [0.0]
  Determination of inspection and maintenance policies constitutes a complex optimization problem. In this work, these challenges are addressed within a joint framework of constrained Partially Observable Decision Processes (POMDP) and multi-agent Deep Reinforcement Learning (DRL) The proposed framework is found to outperform well-established policy baselines and facilitate adept prescription of inspection and intervention actions.
  arXiv  Detail & Related papers  (2020-07-02T20:44:07Z)
• Inverse Active Sensing: Modeling and Understanding Timely Decision-Making [111.07204912245841]
  We develop a framework for the general setting of evidence-based decision-making under endogenous, context-dependent time pressure. We demonstrate how it enables modeling intuitive notions of surprise, suspense, and optimality in decision strategies.
  arXiv  Detail & Related papers  (2020-06-25T02:30:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.