RGMComm: Return Gap Minimization via Discrete Communications in Multi-Agent Reinforcement Learning

• URL: http://arxiv.org/abs/2308.03358v5
• Date: Mon, 18 Dec 2023 20:20:19 GMT
• Title: RGMComm: Return Gap Minimization via Discrete Communications in Multi-Agent Reinforcement Learning
• Authors: Jingdi Chen, Tian Lan, Carlee Joe-Wong
• Abstract summary: Communication is crucial for solving cooperative Multi-Agent Reinforcement Learning tasks in partially observable Markov Decision Processes. We propose the Return-Gap-Minimization Communication (RGMComm) algorithm, which is a surprisingly simple design of discrete message generation functions. Evaluations show that RGMComm significantly outperforms state-of-the-art multi-agent communication baselines.
• Score: 33.86277578441437
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Communication is crucial for solving cooperative Multi-Agent Reinforcement Learning tasks in partially observable Markov Decision Processes. Existing works often rely on black-box methods to encode local information/features into messages shared with other agents, leading to the generation of continuous messages with high communication overhead and poor interpretability. Prior attempts at discrete communication methods generate one-hot vectors trained as part of agents' actions and use the Gumbel softmax operation for calculating message gradients, which are all heuristic designs that do not provide any quantitative guarantees on the expected return. This paper establishes an upper bound on the return gap between an ideal policy with full observability and an optimal partially observable policy with discrete communication. This result enables us to recast multi-agent communication into a novel online clustering problem over the local observations at each agent, with messages as cluster labels and the upper bound on the return gap as clustering loss. To minimize the return gap, we propose the Return-Gap-Minimization Communication (RGMComm) algorithm, which is a surprisingly simple design of discrete message generation functions and is integrated with reinforcement learning through the utilization of a novel Regularized Information Maximization loss function, which incorporates cosine-distance as the clustering metric. Evaluations show that RGMComm significantly outperforms state-of-the-art multi-agent communication baselines and can achieve nearly optimal returns with few-bit messages that are naturally interpretable.

Related papers

• Communication Learning in Multi-Agent Systems from Graph Modeling Perspective [62.13508281188895]
  We introduce a novel approach wherein we conceptualize the communication architecture among agents as a learnable graph. We introduce a temporal gating mechanism for each agent, enabling dynamic decisions on whether to receive shared information at a given time.
  arXiv  Detail & Related papers  (2024-11-01T05:56:51Z)
• Learning Multi-Agent Communication from Graph Modeling Perspective [62.13508281188895]
  We introduce a novel approach wherein we conceptualize the communication architecture among agents as a learnable graph. Our proposed approach, CommFormer, efficiently optimize the communication graph and concurrently refines architectural parameters through gradient descent in an end-to-end manner.
  arXiv  Detail & Related papers  (2024-05-14T12:40:25Z)
• Asynchronous Message-Passing and Zeroth-Order Optimization Based Distributed Learning with a Use-Case in Resource Allocation in Communication Networks [11.182443036683225]
  Distributed learning and adaptation have received significant interest and found wide-ranging applications in machine learning signal processing. This paper specifically focuses on a scenario where agents collaborate towards a common task. Agents, acting as transmitters, collaboratively train their individual policies to maximize a global reward.
  arXiv  Detail & Related papers  (2023-11-08T11:12:27Z)
• Generative AI-aided Joint Training-free Secure Semantic Communications via Multi-modal Prompts [89.04751776308656]
  This paper proposes a GAI-aided SemCom system with multi-model prompts for accurate content decoding. In response to security concerns, we introduce the application of covert communications aided by a friendly jammer.
  arXiv  Detail & Related papers  (2023-09-05T23:24:56Z)
• Communication-Efficient Zeroth-Order Distributed Online Optimization: Algorithm, Theory, and Applications [9.045332526072828]
  This paper focuses on a multi-agent zeroth-order online optimization problem in a federated learning setting for target tracking. The proposed solution is further analyzed in terms of errors and errors in two relevant applications.
  arXiv  Detail & Related papers  (2023-06-09T03:51:45Z)
• Compressed Regression over Adaptive Networks [58.79251288443156]
  We derive the performance achievable by a network of distributed agents that solve, adaptively and in the presence of communication constraints, a regression problem. We devise an optimized allocation strategy where the parameters necessary for the optimization can be learned online by the agents.
  arXiv  Detail & Related papers  (2023-04-07T13:41:08Z)
• Efficient Communication via Self-supervised Information Aggregation for Online and Offline Multi-agent Reinforcement Learning [12.334522644561591]
  We argue that efficient message aggregation is essential for good coordination in cooperative Multi-Agent Reinforcement Learning (MARL) We propose Multi-Agent communication via Self-supervised Information Aggregation (MASIA), where agents can aggregate the received messages into compact representations with high relevance to augment the local policy. We build offline benchmarks for multi-agent communication, which is the first as we know.
  arXiv  Detail & Related papers  (2023-02-19T16:02:16Z)
• Optimization of Image Transmission in a Cooperative Semantic Communication Networks [68.2233384648671]
  A semantic communication framework for image transmission is developed. Servers cooperatively transmit images to a set of users utilizing semantic communication techniques. A multimodal metric is proposed to measure the correlation between the extracted semantic information and the original image.
  arXiv  Detail & Related papers  (2023-01-01T15:59:13Z)
• Multi-agent Communication with Graph Information Bottleneck under Limited Bandwidth (a position paper) [92.11330289225981]
  In many real-world scenarios, communication can be expensive and the bandwidth of the multi-agent system is subject to certain constraints. Redundant messages who occupy the communication resources can block the transmission of informative messages and thus jeopardize the performance. We propose a novel multi-agent communication module, CommGIB, which effectively compresses the structure information and node information in the communication graph to deal with bandwidth-constrained settings.
  arXiv  Detail & Related papers  (2021-12-20T07:53:44Z)
• Learning-based Measurement Scheduling for Loosely-Coupled Cooperative Localization [3.616948583169635]
  In cooperative localization, communicating mobile agents use inter-agent relative measurements to improve their dead-reckoning-based global localization. Measurement scheduling enables an agent to decide which subset of available inter-agent relative measurements it should process when its computational resources are limited. This paper proposes a measurement scheduling for CL that follows the sequential computation approach but reduces the communication and cost by using a neural network-based surrogate model as a proxy for the SG's merit function.
  arXiv  Detail & Related papers  (2021-12-06T08:06:29Z)
• Task-Oriented Communication for Multi-Device Cooperative Edge Inference [14.249444124834719]
  cooperative edge inference can overcome the limited sensing capability of a single device, but it substantially increases the communication overhead and may incur excessive latency. We propose a learning-based communication scheme that optimize local feature extraction and distributed feature encoding in a task-oriented manner.
  arXiv  Detail & Related papers  (2021-09-01T03:56:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.