TAPE: Leveraging Agent Topology for Cooperative Multi-Agent Policy Gradient

• URL: http://arxiv.org/abs/2312.15667v3
• Date: Mon, 15 Jan 2024 09:06:53 GMT
• Title: TAPE: Leveraging Agent Topology for Cooperative Multi-Agent Policy Gradient
• Authors: Xingzhou Lou, Junge Zhang, Timothy J. Norman, Kaiqi Huang, Yali Du
• Abstract summary: We propose an agent topology framework, which decides whether other agents should be considered in policy. Agents can use coalition utility as learning objective instead of global utility by centralized critics or local utility by individual critics. We prove the policy improvement theorem for TAPE and give a theoretical explanation for the improved cooperation among agents.
• Score: 36.83464785085713
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multi-Agent Policy Gradient (MAPG) has made significant progress in recent years. However, centralized critics in state-of-the-art MAPG methods still face the centralized-decentralized mismatch (CDM) issue, which means sub-optimal actions by some agents will affect other agent's policy learning. While using individual critics for policy updates can avoid this issue, they severely limit cooperation among agents. To address this issue, we propose an agent topology framework, which decides whether other agents should be considered in policy gradient and achieves compromise between facilitating cooperation and alleviating the CDM issue. The agent topology allows agents to use coalition utility as learning objective instead of global utility by centralized critics or local utility by individual critics. To constitute the agent topology, various models are studied. We propose Topology-based multi-Agent Policy gradiEnt (TAPE) for both stochastic and deterministic MAPG methods. We prove the policy improvement theorem for stochastic TAPE and give a theoretical explanation for the improved cooperation among agents. Experiment results on several benchmarks show the agent topology is able to facilitate agent cooperation and alleviate CDM issue respectively to improve performance of TAPE. Finally, multiple ablation studies and a heuristic graph search algorithm are devised to show the efficacy of the agent topology.

Related papers

• Deep Multi-Agent Reinforcement Learning for Decentralized Active Hypothesis Testing [11.639503711252663]
  We tackle the multi-agent active hypothesis testing (AHT) problem by introducing a novel algorithm rooted in the framework of deep multi-agent reinforcement learning. We present a comprehensive set of experimental results that effectively showcase the agents' ability to learn collaborative strategies and enhance performance.
  arXiv  Detail & Related papers  (2023-09-14T01:18:04Z)
• SACHA: Soft Actor-Critic with Heuristic-Based Attention for Partially Observable Multi-Agent Path Finding [3.4260993997836753]
  We propose a novel multi-agent actor-critic method called Soft Actor-Critic with Heuristic-Based Attention (SACHA) SACHA learns a neural network for each agent to selectively pay attention to the shortest path guidance from multiple agents within its field of view. We demonstrate decent improvements over several state-of-the-art learning-based MAPF methods with respect to success rate and solution quality.
  arXiv  Detail & Related papers  (2023-07-05T23:36:33Z)
• RACA: Relation-Aware Credit Assignment for Ad-Hoc Cooperation in Multi-Agent Deep Reinforcement Learning [55.55009081609396]
  We propose a novel method, called Relation-Aware Credit Assignment (RACA), which achieves zero-shot generalization in ad-hoc cooperation scenarios. RACA takes advantage of a graph-based encoder relation to encode the topological structure between agents. Our method outperforms baseline methods on the StarCraftII micromanagement benchmark and ad-hoc cooperation scenarios.
  arXiv  Detail & Related papers  (2022-06-02T03:39:27Z)
• Iterated Reasoning with Mutual Information in Cooperative and Byzantine Decentralized Teaming [0.0]
  We show that reformulating an agent's policy to be conditional on the policies of its teammates inherently maximizes Mutual Information (MI) lower-bound when optimizing under Policy Gradient (PG) Our approach, InfoPG, outperforms baselines in learning emergent collaborative behaviors and sets the state-of-the-art in decentralized cooperative MARL tasks.
  arXiv  Detail & Related papers  (2022-01-20T22:54:32Z)
• Tesseract: Tensorised Actors for Multi-Agent Reinforcement Learning [92.05556163518999]
  MARL exacerbates matters by imposing various constraints on communication and observability. For value-based methods, it poses challenges in accurately representing the optimal value function. For policy gradient methods, it makes training the critic difficult and exacerbates the problem of the lagging critic. We show that from a learning theory perspective, both problems can be addressed by accurately representing the associated action-value function.
  arXiv  Detail & Related papers  (2021-05-31T23:08:05Z)
• Dealing with Non-Stationarity in Multi-Agent Reinforcement Learning via Trust Region Decomposition [52.06086375833474]
  Non-stationarity is one thorny issue in multi-agent reinforcement learning. We introduce a $delta$-stationarity measurement to explicitly model the stationarity of a policy sequence. We propose a trust region decomposition network based on message passing to estimate the joint policy divergence.
  arXiv  Detail & Related papers  (2021-02-21T14:46:50Z)
• Off-Policy Multi-Agent Decomposed Policy Gradients [30.389041305278045]
  We investigate causes that hinder the performance of MAPG algorithms and present a multi-agent decomposed policy gradient method (DOP) DOP supports efficient off-policy learning and addresses the issue of centralized-decentralized mismatch and credit assignment. In addition, empirical evaluations on the StarCraft II micromanagement benchmark and multi-agent particle environments demonstrate that DOP significantly outperforms both state-of-the-art value-based and policy-based multi-agent reinforcement learning algorithms.
  arXiv  Detail & Related papers  (2020-07-24T02:21:55Z)
• FACMAC: Factored Multi-Agent Centralised Policy Gradients [103.30380537282517]
  We propose FACtored Multi-Agent Centralised policy gradients (FACMAC) It is a new method for cooperative multi-agent reinforcement learning in both discrete and continuous action spaces. We evaluate FACMAC on variants of the multi-agent particle environments, a novel multi-agent MuJoCo benchmark, and a challenging set of StarCraft II micromanagement tasks.
  arXiv  Detail & Related papers  (2020-03-14T21:29:09Z)
• Multi-Agent Interactions Modeling with Correlated Policies [53.38338964628494]
  In this paper, we cast the multi-agent interactions modeling problem into a multi-agent imitation learning framework. We develop a Decentralized Adrial Imitation Learning algorithm with Correlated policies (CoDAIL) Various experiments demonstrate that CoDAIL can better regenerate complex interactions close to the demonstrators.
  arXiv  Detail & Related papers  (2020-01-04T17:31:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.