Approximate Control for Continuous-Time POMDPs

• URL: http://arxiv.org/abs/2402.01431v2
• Date: Thu, 29 Feb 2024 09:36:17 GMT
• Title: Approximate Control for Continuous-Time POMDPs
• Authors: Yannick Eich, Bastian Alt, Heinz Koeppl
• Abstract summary: This work proposes a decision-making framework for partially observable systems in continuous time with discrete state and action spaces. We employ approximation methods for the filtering and the control problem that scale well with an increasing number of states. We demonstrate the effectiveness of our approach on several partially observed systems, including queueing systems and chemical reaction networks.
• Score: 35.26411026381803
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This work proposes a decision-making framework for partially observable systems in continuous time with discrete state and action spaces. As optimal decision-making becomes intractable for large state spaces we employ approximation methods for the filtering and the control problem that scale well with an increasing number of states. Specifically, we approximate the high-dimensional filtering distribution by projecting it onto a parametric family of distributions, and integrate it into a control heuristic based on the fully observable system to obtain a scalable policy. We demonstrate the effectiveness of our approach on several partially observed systems, including queueing systems and chemical reaction networks.

Related papers

• Arrival Control in Quasi-Reversible Queueing Systems: Optimization and Reinforcement Learning [0.4143603294943439]
  We introduce a versatile scheme for optimizing the arrival rates of quasi-reversible queueing systems.<n>We prove that supplementing a quasi-reversible queueing system with a balanced arrival-control policy preserves the quasi-reversibility.<n>We focus on the problem of admission control and leverage our results in the frameworks of optimization and reinforcement learning.
  arXiv  Detail & Related papers  (2025-05-22T08:05:36Z)
• Latent feedback control of distributed systems in multiple scenarios through deep learning-based reduced order models [3.5161229331588095]
  Continuous monitoring and real-time control of high-dimensional distributed systems are crucial in applications to ensure a desired physical behavior. Traditional feedback control design that relies on full-order models fails to meet these requirements due to the delay in the control computation. We propose a real-time closed-loop control strategy enhanced by nonlinear non-intrusive Deep Learning-based Reduced Order Models (DL-ROMs)
  arXiv  Detail & Related papers  (2024-12-13T08:04:21Z)
• Locally Interdependent Multi-Agent MDP: Theoretical Framework for Decentralized Agents with Dynamic Dependencies [6.015898117103069]
  We analyze a decentralized model with dynamically varying dependencies called the Locally Interdependent Multi-Agent MDP. Despite the intractability that general partially observable multi-agent systems suffer from, we propose three closed-form policies.
  arXiv  Detail & Related papers  (2024-06-10T22:11:00Z)
• Many-body entropies and entanglement from polynomially-many local measurements [0.26388783516590225]
  We show that efficient estimation strategies exist under the assumption that all the spatial correlation lengths are finite. We argue that our method could be practically useful to detect bipartite mixed-state entanglement for large numbers of qubits available in today's quantum platforms.
  arXiv  Detail & Related papers  (2023-11-14T12:13:15Z)
• Optimal Control of Logically Constrained Partially Observable and Multi-Agent Markov Decision Processes [5.471640959988549]
  We first introduce an optimal control theory for partially observable Markov decision processes. We provide a structured methodology for synthesizing policies that maximize a cumulative reward. We then build on this approach to design an optimal control framework for logically constrained multi-agent settings.
  arXiv  Detail & Related papers  (2023-05-24T05:15:36Z)
• In-Distribution Barrier Functions: Self-Supervised Policy Filters that Avoid Out-of-Distribution States [84.24300005271185]
  We propose a control filter that wraps any reference policy and effectively encourages the system to stay in-distribution with respect to offline-collected safe demonstrations. Our method is effective for two different visuomotor control tasks in simulation environments, including both top-down and egocentric view settings.
  arXiv  Detail & Related papers  (2023-01-27T22:28:19Z)
• Computationally Efficient PAC RL in POMDPs with Latent Determinism and Conditional Embeddings [97.12538243736705]
  We study reinforcement learning with function approximation for large-scale Partially Observable Decision Processes (POMDPs) Our algorithm provably scales to large-scale POMDPs.
  arXiv  Detail & Related papers  (2022-06-24T05:13:35Z)
• Finite-Time Analysis of Natural Actor-Critic for POMDPs [29.978816372127085]
  We consider the reinforcement learning problem for partially observed Markov decision processes (POMDPs) with large or even countably infinite state spaces. We consider a natural actor-critic method that employs a finite internal memory for policy parameterization. We show that this error can be made small in the case of sliding-block controllers by using larger block sizes.
  arXiv  Detail & Related papers  (2022-02-20T07:42:00Z)
• Numerical estimation of reachable and controllability sets for a two-level open quantum system driven by coherent and incoherent controls [77.34726150561087]
  The article considers a two-level open quantum system governed by the Gorini--Kossakowski--Lindblad--Sudarshan master equation. The system is analyzed using Bloch parametrization of the system's density matrix.
  arXiv  Detail & Related papers  (2021-06-18T14:23:29Z)
• Correct-by-construction reach-avoid control of partially observable linear stochastic systems [7.912008109232803]
  We formalize a robust feedback controller for reach-avoid control of discrete-time, linear time-invariant systems. The problem is to compute a controller that satisfies the required provestate abstraction problem.
  arXiv  Detail & Related papers  (2021-03-03T13:46:52Z)
• Stein Variational Model Predictive Control [130.60527864489168]
  Decision making under uncertainty is critical to real-world, autonomous systems. Model Predictive Control (MPC) methods have demonstrated favorable performance in practice, but remain limited when dealing with complex distributions. We show that this framework leads to successful planning in challenging, non optimal control problems.
  arXiv  Detail & Related papers  (2020-11-15T22:36:59Z)
• Decentralized MCTS via Learned Teammate Models [89.24858306636816]
  We present a trainable online decentralized planning algorithm based on decentralized Monte Carlo Tree Search. We show that deep learning and convolutional neural networks can be employed to produce accurate policy approximators.
  arXiv  Detail & Related papers  (2020-03-19T13:10:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.