Belief States for Cooperative Multi-Agent Reinforcement Learning under Partial Observability

• URL: http://arxiv.org/abs/2504.08417v1
• Date: Fri, 11 Apr 2025 10:21:58 GMT
• Title: Belief States for Cooperative Multi-Agent Reinforcement Learning under Partial Observability
• Authors: Paul J. Pritz, Kin K. Leung,
• Abstract summary: We propose the use of learned beliefs on the underlying state of the system to overcome challenges in reinforcement learning.<n>We create an end-to-end model for cooperative multi-agent reinforcement learning under partial observability.<n>We evaluate our proposed method on diverse partially observable multi-agent tasks designed to exhibit different variants of partial observability.
• Score: 3.2912049028407897
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reinforcement learning in partially observable environments is typically challenging, as it requires agents to learn an estimate of the underlying system state. These challenges are exacerbated in multi-agent settings, where agents learn simultaneously and influence the underlying state as well as each others' observations. We propose the use of learned beliefs on the underlying state of the system to overcome these challenges and enable reinforcement learning with fully decentralized training and execution. Our approach leverages state information to pre-train a probabilistic belief model in a self-supervised fashion. The resulting belief states, which capture both inferred state information as well as uncertainty over this information, are then used in a state-based reinforcement learning algorithm to create an end-to-end model for cooperative multi-agent reinforcement learning under partial observability. By separating the belief and reinforcement learning tasks, we are able to significantly simplify the policy and value function learning tasks and improve both the convergence speed and the final performance. We evaluate our proposed method on diverse partially observable multi-agent tasks designed to exhibit different variants of partial observability.

Related papers

• Learning Interpretable Policies in Hindsight-Observable POMDPs through Partially Supervised Reinforcement Learning [57.67629402360924]
  We introduce the Partially Supervised Reinforcement Learning (PSRL) framework. At the heart of PSRL is the fusion of both supervised and unsupervised learning. We show that PSRL offers a potent balance, enhancing model interpretability while preserving, and often significantly outperforming, the performance benchmarks set by traditional methods.
  arXiv  Detail & Related papers  (2024-02-14T16:23:23Z)
• Provable Representation with Efficient Planning for Partial Observable Reinforcement Learning [74.67655210734338]
  In most real-world reinforcement learning applications, state information is only partially observable, which breaks the Markov decision process assumption. We develop a representation-based perspective that leads to a coherent framework and tractable algorithmic approach for practical reinforcement learning from partial observations. We empirically demonstrate the proposed algorithm can surpass state-of-the-art performance with partial observations across various benchmarks.
  arXiv  Detail & Related papers  (2023-11-20T23:56:58Z)
• Learning Unseen Modality Interaction [54.23533023883659]
  Multimodal learning assumes all modality combinations of interest are available during training to learn cross-modal correspondences. We pose the problem of unseen modality interaction and introduce a first solution. It exploits a module that projects the multidimensional features of different modalities into a common space with rich information preserved.
  arXiv  Detail & Related papers  (2023-06-22T10:53:10Z)
• Sample-efficient Adversarial Imitation Learning [45.400080101596956]
  We propose a self-supervised representation-based adversarial imitation learning method to learn state and action representations. We show a 39% relative improvement over existing adversarial imitation learning methods on MuJoCo in a setting limited to 100 expert state-action pairs.
  arXiv  Detail & Related papers  (2023-03-14T12:36:01Z)
• Learning Generalizable Representations for Reinforcement Learning via Adaptive Meta-learner of Behavioral Similarities [43.327357653393015]
  We propose a novel meta-learner-based framework for representation learning regarding behavioral similarities for reinforcement learning. We empirically demonstrate that our proposed framework outperforms state-of-the-art baselines on several benchmarks.
  arXiv  Detail & Related papers  (2022-12-26T11:11:23Z)
• Imitation Learning by State-Only Distribution Matching [2.580765958706854]
  Imitation Learning from observation describes policy learning in a similar way to human learning. We propose a non-adversarial learning-from-observations approach, together with an interpretable convergence and performance metric.
  arXiv  Detail & Related papers  (2022-02-09T08:38:50Z)
• Learning State Representations via Retracing in Reinforcement Learning [25.755855290244103]
  Learning via retracing is a self-supervised approach for learning the state representation for reinforcement learning tasks. We introduce Cycle-Consistency World Model (CCWM), a concrete instantiation of learning via retracing. We show that CCWM achieves state-of-the-art performance in terms of sample efficiency and performance.
  arXiv  Detail & Related papers  (2021-11-24T16:19:59Z)
• Visual Adversarial Imitation Learning using Variational Models [60.69745540036375]
  Reward function specification remains a major impediment for learning behaviors through deep reinforcement learning. Visual demonstrations of desired behaviors often presents an easier and more natural way to teach agents. We develop a variational model-based adversarial imitation learning algorithm.
  arXiv  Detail & Related papers  (2021-07-16T00:15:18Z)
• Multi-Agent Imitation Learning with Copulas [102.27052968901894]
  Multi-agent imitation learning aims to train multiple agents to perform tasks from demonstrations by learning a mapping between observations and actions. In this paper, we propose to use copula, a powerful statistical tool for capturing dependence among random variables, to explicitly model the correlation and coordination in multi-agent systems. Our proposed model is able to separately learn marginals that capture the local behavioral patterns of each individual agent, as well as a copula function that solely and fully captures the dependence structure among agents.
  arXiv  Detail & Related papers  (2021-07-10T03:49:41Z)
• Off-policy Evaluation in Infinite-Horizon Reinforcement Learning with Latent Confounders [62.54431888432302]
  We study an OPE problem in an infinite-horizon, ergodic Markov decision process with unobserved confounders. We show how, given only a latent variable model for states and actions, policy value can be identified from off-policy data.
  arXiv  Detail & Related papers  (2020-07-27T22:19:01Z)
• Efficient Empowerment Estimation for Unsupervised Stabilization [75.32013242448151]
  empowerment principle enables unsupervised stabilization of dynamical systems at upright positions. We propose an alternative solution based on a trainable representation of a dynamical system as a Gaussian channel. We show that our method has a lower sample complexity, is more stable in training, possesses the essential properties of the empowerment function, and allows estimation of empowerment from images.
  arXiv  Detail & Related papers  (2020-07-14T21:10:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.