Multi-Environment POMDPs: Discrete Model Uncertainty Under Partial Observability

• URL: http://arxiv.org/abs/2510.23744v1
• Date: Mon, 27 Oct 2025 18:24:11 GMT
• Title: Multi-Environment POMDPs: Discrete Model Uncertainty Under Partial Observability
• Authors: Eline M. Bovy, Caleb Probine, Marnix Suilen, Ufuk Topcu, Nils Jansen,
• Abstract summary: Multi-environment POMDPs (ME-POMDPs) extend standard POMDPs with discrete model uncertainty.<n>We show that ME-POMDPs can be generalized to POMDPs with sets of initial beliefs.<n>We then devise exact and approximate (point-based) algorithms to compute robust policies for AB-POMDPs.
• Score: 29.63953552645502
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multi-environment POMDPs (ME-POMDPs) extend standard POMDPs with discrete model uncertainty. ME-POMDPs represent a finite set of POMDPs that share the same state, action, and observation spaces, but may arbitrarily vary in their transition, observation, and reward models. Such models arise, for instance, when multiple domain experts disagree on how to model a problem. The goal is to find a single policy that is robust against any choice of POMDP within the set, i.e., a policy that maximizes the worst-case reward across all POMDPs. We generalize and expand on existing work in the following way. First, we show that ME-POMDPs can be generalized to POMDPs with sets of initial beliefs, which we call adversarial-belief POMDPs (AB-POMDPs). Second, we show that any arbitrary ME-POMDP can be reduced to a ME-POMDP that only varies in its transition and reward functions or only in its observation and reward functions, while preserving (optimal) policies. We then devise exact and approximate (point-based) algorithms to compute robust policies for AB-POMDPs, and thus ME-POMDPs. We demonstrate that we can compute policies for standard POMDP benchmarks extended to the multi-environment setting.

Related papers

• Learning Policy Representations for Steerable Behavior Synthesis [80.4542176039074]
  Given a Markov decision process (MDP), we seek to learn representations for a range of policies to facilitate behavior steering at test time.<n>We show that these representations can be approximated uniformly for a range of policies using a set-based architecture.<n>We use variational generative approach to induce a smooth latent space, and further shape it with contrastive learning so that latent distances align with differences in value functions.
  arXiv  Detail & Related papers  (2026-01-29T21:52:06Z)
• Scalable Policy-Based RL Algorithms for POMDPs [6.2229686397601585]
  We consider an approach that solves a Partially Observable Reinforcement Learning (PORL) problem by approximating a POMDP model into a finite-state Markov Decision Process (MDP)<n>We show that the approximation error decreases exponentially with the length of this history.<n>To the best of our knowledge, our finite-time bounds are the first to explicitly quantify the error introduced when applying standard TD learning to a setting where the true dynamics are not Markovian.
  arXiv  Detail & Related papers  (2025-10-08T00:33:38Z)
• Robust Finite-Memory Policy Gradients for Hidden-Model POMDPs [7.447371788025412]
  Partially observable Markov decision processes (POMDPs) model specific environments in sequential decision-making under uncertainty.<n>We show that HM-POMDPs capture sets of different environment models, that is, POMDPs with a shared action and observation space.<n>A policy is robust for a given HM-POMDP if it achieves sufficient performance for each of its POMDPs.
  arXiv  Detail & Related papers  (2025-05-14T16:15:58Z)
• Optimality Guarantees for Particle Belief Approximation of POMDPs [55.83001584645448]
  Partially observable Markov decision processes (POMDPs) provide a flexible representation for real-world decision and control problems. POMDPs are notoriously difficult to solve, especially when the state and observation spaces are continuous or hybrid. We propose a theory characterizing the approximation error of the particle filtering techniques that these algorithms use.
  arXiv  Detail & Related papers  (2022-10-10T21:11:55Z)
• Towards Using Fully Observable Policies for POMDPs [0.0]
  Partially Observable Markov Decision Process (POMDP) is a framework applicable to many real world problems. We propose an approach to solve POMDPs with multimodal belief by relying on a policy that solves the fully observable version.
  arXiv  Detail & Related papers  (2022-07-24T13:22:13Z)
• Twice regularized MDPs and the equivalence between robustness and regularization [65.58188361659073]
  We show that policy iteration on reward-robust MDPs can have the same time complexity as on regularized MDPs. We generalize regularized MDPs to twice regularized MDPs.
  arXiv  Detail & Related papers  (2021-10-12T18:33:45Z)
• Safe Exploration by Solving Early Terminated MDP [77.10563395197045]
  We introduce a new approach to address safe RL problems under the framework of Early TerminatedP (ET-MDP) We first define the ET-MDP as an unconstrained algorithm with the same optimal value function as its corresponding CMDP. An off-policy algorithm based on context models is then proposed to solve the ET-MDP, which thereby solves the corresponding CMDP with better performance and improved learning efficiency.
  arXiv  Detail & Related papers  (2021-07-09T04:24:40Z)
• Permutation Invariant Policy Optimization for Mean-Field Multi-Agent Reinforcement Learning: A Principled Approach [128.62787284435007]
  We propose the mean-field proximal policy optimization (MF-PPO) algorithm, at the core of which is a permutation-invariant actor-critic neural architecture. We prove that MF-PPO attains the globally optimal policy at a sublinear rate of convergence. In particular, we show that the inductive bias introduced by the permutation-invariant neural architecture enables MF-PPO to outperform existing competitors.
  arXiv  Detail & Related papers  (2021-05-18T04:35:41Z)
• Modular Deep Reinforcement Learning for Continuous Motion Planning with Temporal Logic [59.94347858883343]
  This paper investigates the motion planning of autonomous dynamical systems modeled by Markov decision processes (MDP) The novelty is to design an embedded product MDP (EP-MDP) between the LDGBA and the MDP. The proposed LDGBA-based reward shaping and discounting schemes for the model-free reinforcement learning (RL) only depend on the EP-MDP states.
  arXiv  Detail & Related papers  (2021-02-24T01:11:25Z)
• A Relation Analysis of Markov Decision Process Frameworks [26.308541799686505]
  We study the relation between different Decision Process (MDP) frameworks in the machine learning and econometrics literature. We show that the entropy-regularized MDP is equivalent to a MDP model, and is strictly subsumed by the general regularized MDP.
  arXiv  Detail & Related papers  (2020-08-18T09:27:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.