Deep Meta Coordination Graphs for Multi-agent Reinforcement Learning

• URL: http://arxiv.org/abs/2502.04028v1
• Date: Thu, 06 Feb 2025 12:35:52 GMT
• Title: Deep Meta Coordination Graphs for Multi-agent Reinforcement Learning
• Authors: Nikunj Gupta, James Zachary Hare, Rajgopal Kannan, Viktor Prasanna,
• Abstract summary: Deep meta coordination graphs (DMCG) for learning cooperative policies in multi-agent reinforcement learning (MARL) DMCG captures useful higher-order and indirect relationships among agents. It then employs a graph convolutional network module to learn powerful representations in an end-to-end manner.
• Score: 2.650735171795961
• License:
• Abstract: This paper presents deep meta coordination graphs (DMCG) for learning cooperative policies in multi-agent reinforcement learning (MARL). Coordination graph formulations encode local interactions and accordingly factorize the joint value function of all agents to improve efficiency in MARL. However, existing approaches rely solely on pairwise relations between agents, which potentially oversimplifies complex multi-agent interactions. DMCG goes beyond these simple direct interactions by also capturing useful higher-order and indirect relationships among agents. It generates novel graph structures accommodating multiple types of interactions and arbitrary lengths of multi-hop connections in coordination graphs to model such interactions. It then employs a graph convolutional network module to learn powerful representations in an end-to-end manner. We demonstrate its effectiveness in multiple coordination problems in MARL where other state-of-the-art methods can suffer from sample inefficiency or fail entirely. All codes can be found here: https://github.com/Nikunj-Gupta/dmcg-marl.

Related papers

• Communication Learning in Multi-Agent Systems from Graph Modeling Perspective [62.13508281188895]
  We introduce a novel approach wherein we conceptualize the communication architecture among agents as a learnable graph. We introduce a temporal gating mechanism for each agent, enabling dynamic decisions on whether to receive shared information at a given time.
  arXiv  Detail & Related papers  (2024-11-01T05:56:51Z)
• Scaling Large-Language-Model-based Multi-Agent Collaboration [75.5241464256688]
  Pioneering advancements in large language model-powered agents have underscored the design pattern of multi-agent collaboration. Inspired by the neural scaling law, this study investigates whether a similar principle applies to increasing agents in multi-agent collaboration.
  arXiv  Detail & Related papers  (2024-06-11T11:02:04Z)
• Learning Multi-Agent Communication from Graph Modeling Perspective [62.13508281188895]
  We introduce a novel approach wherein we conceptualize the communication architecture among agents as a learnable graph. Our proposed approach, CommFormer, efficiently optimize the communication graph and concurrently refines architectural parameters through gradient descent in an end-to-end manner.
  arXiv  Detail & Related papers  (2024-05-14T12:40:25Z)
• Self-Clustering Hierarchical Multi-Agent Reinforcement Learning with Extensible Cooperation Graph [9.303181273699417]
  This paper proposes a novel hierarchical MARL model called Hierarchical Cooperation Graph Learning (HCGL) HCGL has three components: a dynamic Cooperation Graph (ECG) for achieving self-clustering cooperation; a group of graph operators for adjusting the topology of ECG; and an MARL for training these graph operators. In our experiments, the HCGL model has shown outstanding performance in multi-agent benchmarks with sparse rewards.
  arXiv  Detail & Related papers  (2024-03-26T19:19:16Z)
• MADiff: Offline Multi-agent Learning with Diffusion Models [79.18130544233794]
  MADiff is a diffusion-based multi-agent learning framework. It works as both a decentralized policy and a centralized controller. Our experiments demonstrate that MADiff outperforms baseline algorithms across various multi-agent learning tasks.
  arXiv  Detail & Related papers  (2023-05-27T02:14:09Z)
• Unrolled Graph Learning for Multi-Agent Collaboration [37.239120967721156]
  We propose a distributed multi-agent learning model inspired by human collaboration. Agents can autonomously detect suitable collaborators and refer to collaborators' model for better performance.
  arXiv  Detail & Related papers  (2022-10-31T07:05:44Z)
• A Cooperation Graph Approach for Multiagent Sparse Reward Reinforcement Learning [7.2972297703292135]
  Multiagent reinforcement learning (MARL) can solve complex cooperative tasks. In this paper, we design a graph network called Cooperation Graph (CG) We propose a Cooperation Graph Multiagent Reinforcement Learning (CG-MARL) algorithm, which can efficiently deal with the sparse reward problem in multiagent tasks.
  arXiv  Detail & Related papers  (2022-08-05T06:32:16Z)
• Self-Organized Polynomial-Time Coordination Graphs [21.02670428540549]
  Coordination graph is a promising approach to model agent collaboration in reinforcement learning. One challenge in this paradigm is the complexity of computing maximum-value actions for a graph-based value factorization. This paper proposes a novel method, named Self-Organized Polynomial-time Coordination Graphs (SOP-CG), which uses structured graph classes to guarantee the optimality of the induced DCOPs.
  arXiv  Detail & Related papers  (2021-12-07T07:42:40Z)
• Cooperative Policy Learning with Pre-trained Heterogeneous Observation Representations [51.8796674904734]
  We propose a new cooperative learning framework with pre-trained heterogeneous observation representations. We employ an encoder-decoder based graph attention to learn the intricate interactions and heterogeneous representations.
  arXiv  Detail & Related papers  (2020-12-24T04:52:29Z)
• Multi-Agent Interactions Modeling with Correlated Policies [53.38338964628494]
  In this paper, we cast the multi-agent interactions modeling problem into a multi-agent imitation learning framework. We develop a Decentralized Adrial Imitation Learning algorithm with Correlated policies (CoDAIL) Various experiments demonstrate that CoDAIL can better regenerate complex interactions close to the demonstrators.
  arXiv  Detail & Related papers  (2020-01-04T17:31:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.