Heterogeneous Multi-Agent Reinforcement Learning for Zero-Shot Scalable Collaboration

• URL: http://arxiv.org/abs/2404.03869v1
• Date: Fri, 5 Apr 2024 03:02:57 GMT
• Title: Heterogeneous Multi-Agent Reinforcement Learning for Zero-Shot Scalable Collaboration
• Authors: Xudong Guo, Daming Shi, Junjie Yu, Wenhui Fan,
• Abstract summary: We propose a novel framework named scalable and Heterogeneous Proximal Policy Optimization (SHPPO) Our approach is based on the state-of-the-art backbone PPO-based algorithm as SHPPO. SHPPO exhibits superior performance over the baselines in classic MARL environments like Starcraft Multi-Agent Challenge (SMAC) and Google Research Football (GRF)
• Score: 5.326588461041464
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The rise of multi-agent systems, especially the success of multi-agent reinforcement learning (MARL), is reshaping our future across diverse domains like autonomous vehicle networks. However, MARL still faces significant challenges, particularly in achieving zero-shot scalability, which allows trained MARL models to be directly applied to unseen tasks with varying numbers of agents. In addition, real-world multi-agent systems usually contain agents with different functions and strategies, while the existing scalable MARL methods only have limited heterogeneity. To address this, we propose a novel MARL framework named Scalable and Heterogeneous Proximal Policy Optimization (SHPPO), integrating heterogeneity into parameter-shared PPO-based MARL networks. we first leverage a latent network to adaptively learn strategy patterns for each agent. Second, we introduce a heterogeneous layer for decision-making, whose parameters are specifically generated by the learned latent variables. Our approach is scalable as all the parameters are shared except for the heterogeneous layer, and gains both inter-individual and temporal heterogeneity at the same time. We implement our approach based on the state-of-the-art backbone PPO-based algorithm as SHPPO, while our approach is agnostic to the backbone and can be seamlessly plugged into any parameter-shared MARL method. SHPPO exhibits superior performance over the baselines such as MAPPO and HAPPO in classic MARL environments like Starcraft Multi-Agent Challenge (SMAC) and Google Research Football (GRF), showcasing enhanced zero-shot scalability and offering insights into the learned latent representation's impact on team performance by visualization.

Related papers

• Adaptability in Multi-Agent Reinforcement Learning: A Framework and Unified Review [9.246912481179464]
  Multi-Agent Reinforcement Learning (MARL) has shown clear effectiveness in coordinating multiple agents across simulated benchmarks and constrained scenarios.<n>This survey contributes to the development of algorithms that are better suited for deployment in dynamic, real-world multi-agent systems.
  arXiv  Detail & Related papers  (2025-07-14T10:39:17Z)
• Graft: Integrating the Domain Knowledge via Efficient Parameter Synergy for MLLMs [56.76586846269894]
  Multimodal Large Language Models (MLLMs) have achieved success across various domains.<n>Despite its importance, the study of knowledge sharing among domain-specific MLLMs remains largely underexplored.<n>We propose a unified parameter integration framework that enables modular composition of expert capabilities.
  arXiv  Detail & Related papers  (2025-06-30T15:07:41Z)
• Multiple Weaks Win Single Strong: Large Language Models Ensemble Weak Reinforcement Learning Agents into a Supreme One [28.264011412168347]
  Model ensemble is a useful approach in reinforcement learning (RL) for training effective agents.<n>We propose LLM-Ens, a novel approach that enhances RL model ensemble with task-specific semantic understandings.
  arXiv  Detail & Related papers  (2025-05-21T09:35:43Z)
• SPECTra: Scalable Multi-Agent Reinforcement Learning with Permutation-Free Networks [3.7687375904925484]
  In cooperative multi-agent reinforcement learning (MARL), the permutation problem where the state space grows exponentially with the number of agents reduces sample efficiency. We propose a novel agent network and a non-linear mixing network that ensure permutation-equivariance and scalability. Our approach achieves superior learning performance compared to existing methods.
  arXiv  Detail & Related papers  (2025-03-14T04:26:51Z)
• Cooperative Multi-Agent Planning with Adaptive Skill Synthesis [16.228784877899976]
  Multi-agent systems with reinforcement learning face challenges in sample efficiency, interpretability, and transferability. We present a novel multi-agent architecture that integrates vision-language models (VLMs) with a dynamic skill library and structured communication for decentralized closed-loop decision-making.
  arXiv  Detail & Related papers  (2025-02-14T13:23:18Z)
• HyperMARL: Adaptive Hypernetworks for Multi-Agent RL [10.00022425344723]
  HyperMARL is a parameter sharing approach that uses hypernetworks to generate agent-specific parameters without altering the learning objective. It consistently performs competitively with fully shared, non- parameter-sharing, and diversity-promoting baselines. These findings establish hypernetworks as a versatile approach for MARL across diverse environments.
  arXiv  Detail & Related papers  (2024-12-05T15:09:51Z)
• AmoebaLLM: Constructing Any-Shape Large Language Models for Efficient and Instant Deployment [13.977849745488339]
  AmoebaLLM is a novel framework designed to enable the instant derivation of large language models of arbitrary shapes. AmoebaLLM significantly facilitates rapid deployment tailored to various platforms and applications.
  arXiv  Detail & Related papers  (2024-11-15T22:02:28Z)
• Improving Global Parameter-sharing in Physically Heterogeneous Multi-agent Reinforcement Learning with Unified Action Space [22.535906675532196]
  In a multi-agent system, action semantics indicates the different influences of agents' actions toward other entities. Previous multi-agent reinforcement learning (MARL) algorithms apply global parameter-sharing across different types of heterogeneous agents. We introduce the Unified Action Space (UAS) to fulfill the requirement.
  arXiv  Detail & Related papers  (2024-08-14T09:15:11Z)
• Decentralized Transformers with Centralized Aggregation are Sample-Efficient Multi-Agent World Models [106.94827590977337]
  We propose a novel world model for Multi-Agent RL (MARL) that learns decentralized local dynamics for scalability. We also introduce a Perceiver Transformer as an effective solution to enable centralized representation aggregation. Results on Starcraft Multi-Agent Challenge (SMAC) show that it outperforms strong model-free approaches and existing model-based methods in both sample efficiency and overall performance.
  arXiv  Detail & Related papers  (2024-06-22T12:40:03Z)
• Efficient Adaptation in Mixed-Motive Environments via Hierarchical Opponent Modeling and Planning [51.52387511006586]
  We propose Hierarchical Opponent modeling and Planning (HOP), a novel multi-agent decision-making algorithm. HOP is hierarchically composed of two modules: an opponent modeling module that infers others' goals and learns corresponding goal-conditioned policies. HOP exhibits superior few-shot adaptation capabilities when interacting with various unseen agents, and excels in self-play scenarios.
  arXiv  Detail & Related papers  (2024-06-12T08:48:06Z)
• A Generic Shared Attention Mechanism for Various Backbone Neural Networks [53.36677373145012]
  Self-attention modules (SAMs) produce strongly correlated attention maps across different layers. Dense-and-Implicit Attention (DIA) shares SAMs across layers and employs a long short-term memory module. Our simple yet effective DIA can consistently enhance various network backbones.
  arXiv  Detail & Related papers  (2022-10-27T13:24:08Z)
• Efficient Model-based Multi-agent Reinforcement Learning via Optimistic Equilibrium Computation [93.52573037053449]
  H-MARL (Hallucinated Multi-Agent Reinforcement Learning) learns successful equilibrium policies after a few interactions with the environment. We demonstrate our approach experimentally on an autonomous driving simulation benchmark.
  arXiv  Detail & Related papers  (2022-03-14T17:24:03Z)
• Permutation Invariant Policy Optimization for Mean-Field Multi-Agent Reinforcement Learning: A Principled Approach [128.62787284435007]
  We propose the mean-field proximal policy optimization (MF-PPO) algorithm, at the core of which is a permutation-invariant actor-critic neural architecture. We prove that MF-PPO attains the globally optimal policy at a sublinear rate of convergence. In particular, we show that the inductive bias introduced by the permutation-invariant neural architecture enables MF-PPO to outperform existing competitors.
  arXiv  Detail & Related papers  (2021-05-18T04:35:41Z)
• Deep Autoencoding Topic Model with Scalable Hybrid Bayesian Inference [55.35176938713946]
  We develop deep autoencoding topic model (DATM) that uses a hierarchy of gamma distributions to construct its multi-stochastic-layer generative network. We propose a Weibull upward-downward variational encoder that deterministically propagates information upward via a deep neural network, followed by a downward generative model. The efficacy and scalability of our models are demonstrated on both unsupervised and supervised learning tasks on big corpora.
  arXiv  Detail & Related papers  (2020-06-15T22:22:56Z)
• Scaling Up Multiagent Reinforcement Learning for Robotic Systems: Learn an Adaptive Sparse Communication Graph [39.48317026356428]
  The complexity of multiagent reinforcement learning increases exponentially with respect to the agent number. One critical feature in MARL that is often neglected is that the interactions between agents are quite sparse. We propose an adaptive sparse attention mechanism by generalizing a sparsity-inducing activation function. We show that our algorithm can learn an interpretable sparse structure and outperforms previous works by a significant margin on applications involving a large-scale multiagent system.
  arXiv  Detail & Related papers  (2020-03-02T17:18:25Z)
• Generalized Hidden Parameter MDPs Transferable Model-based RL in a Handful of Trials [13.051708608864539]
  Generalized Hidden MDPs (GHP-MDPs) describe a family of MDPs where both dynamics and reward can change as a function of hidden parameters that vary across tasks. We experimentally demonstrate state-of-the-art performance and sample-efficiency on a new challenging MuJoCo task using reward and dynamics latent spaces.
  arXiv  Detail & Related papers  (2020-02-08T02:49:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.