Adaptive parameter sharing for multi-agent reinforcement learning

• URL: http://arxiv.org/abs/2312.09009v1
• Date: Thu, 14 Dec 2023 15:00:32 GMT
• Title: Adaptive parameter sharing for multi-agent reinforcement learning
• Authors: Dapeng Li, Na Lou, Bin Zhang, Zhiwei Xu, Guoliang Fan
• Abstract summary: We propose a novel parameter sharing method inspired by research pertaining to the brain in biology. It maps each type of agent to different regions within a shared network based on their identity, resulting in distinctworks. Our method can increase the diversity of strategies among different agents without additional training parameters.
• Score: 16.861543418593044
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Parameter sharing, as an important technique in multi-agent systems, can effectively solve the scalability issue in large-scale agent problems. However, the effectiveness of parameter sharing largely depends on the environment setting. When agents have different identities or tasks, naive parameter sharing makes it difficult to generate sufficiently differentiated strategies for agents. Inspired by research pertaining to the brain in biology, we propose a novel parameter sharing method. It maps each type of agent to different regions within a shared network based on their identity, resulting in distinct subnetworks. Therefore, our method can increase the diversity of strategies among different agents without introducing additional training parameters. Through experiments conducted in multiple environments, our method has shown better performance than other parameter sharing methods.

Related papers

• Kaleidoscope: Learnable Masks for Heterogeneous Multi-agent Reinforcement Learning [14.01772209044574]
  We introduce emphKaleidoscope, a novel adaptive partial parameter sharing scheme. It promotes diversity among policy networks by encouraging discrepancy among these masks, without sacrificing the efficiencies of parameter sharing. We extend Kaleidoscope to critic ensembles in the context of actor-critic algorithms, which could help improve value estimations.
  arXiv  Detail & Related papers  (2024-10-11T05:22:54Z)
• Merging Multi-Task Models via Weight-Ensembling Mixture of Experts [64.94129594112557]
  Merging Transformer-based models trained on different tasks into a single unified model can execute all the tasks concurrently. Previous methods, exemplified by task arithmetic, have been proven to be both effective and scalable. We propose to merge most of the parameters while upscaling the Transformer layers to a weight-ensembling mixture of experts (MoE) module.
  arXiv  Detail & Related papers  (2024-02-01T08:58:57Z)
• Measuring Policy Distance for Multi-Agent Reinforcement Learning [9.80588687020087]
  We propose the multi-agent policy distance (MAPD), a tool for measuring policy differences in multi-agent reinforcement learning (MARL) By learning the conditional representations of agents' decisions, MAPD can compute the policy distance between any pair of agents. We also extend MAPD to a customizable version, which can quantify differences among agent policies on specified aspects.
  arXiv  Detail & Related papers  (2024-01-20T15:34:51Z)
• Parameter Sharing with Network Pruning for Scalable Multi-Agent Deep Reinforcement Learning [20.35644044703191]
  We propose a simple method that adopts structured pruning for a deep neural network to increase the representational capacity of the joint policy without introducing additional parameters. We evaluate the proposed method on several benchmark tasks, and numerical results show that the proposed method significantly outperforms other parameter-sharing methods.
  arXiv  Detail & Related papers  (2023-03-02T02:17:14Z)
• Multi-agent Deep Covering Skill Discovery [50.812414209206054]
  We propose Multi-agent Deep Covering Option Discovery, which constructs the multi-agent options through minimizing the expected cover time of the multiple agents' joint state space. Also, we propose a novel framework to adopt the multi-agent options in the MARL process. We show that the proposed algorithm can effectively capture the agent interactions with the attention mechanism, successfully identify multi-agent options, and significantly outperforms prior works using single-agent options or no options.
  arXiv  Detail & Related papers  (2022-10-07T00:40:59Z)
• Policy Diagnosis via Measuring Role Diversity in Cooperative Multi-agent RL [107.58821842920393]
  We quantify the agent's behavior difference and build its relationship with the policy performance via bf Role Diversity We find that the error bound in MARL can be decomposed into three parts that have a strong relation to the role diversity. The decomposed factors can significantly impact policy optimization on three popular directions.
  arXiv  Detail & Related papers  (2022-06-01T04:58:52Z)
• LDSA: Learning Dynamic Subtask Assignment in Cooperative Multi-Agent Reinforcement Learning [122.47938710284784]
  We propose a novel framework for learning dynamic subtask assignment (LDSA) in cooperative MARL. To reasonably assign agents to different subtasks, we propose an ability-based subtask selection strategy. We show that LDSA learns reasonable and effective subtask assignment for better collaboration.
  arXiv  Detail & Related papers  (2022-05-05T10:46:16Z)
• Scaling Multi-Agent Reinforcement Learning with Selective Parameter Sharing [4.855663359344748]
  Sharing parameters in deep reinforcement learning has played an essential role in allowing algorithms to scale to a large number of agents. However, having all agents share the same parameters can also have a detrimental effect on learning. We propose a novel method to automatically identify agents which may benefit from sharing parameters by partitioning them based on their abilities and goals.
  arXiv  Detail & Related papers  (2021-02-15T11:33:52Z)
• Small Towers Make Big Differences [59.243296878666285]
  Multi-task learning aims at solving multiple machine learning tasks at the same time. A good solution to a multi-task learning problem should be generalizable in addition to being Pareto optimal. We propose a method of under- parameterized self-auxiliaries for multi-task models to achieve the best of both worlds.
  arXiv  Detail & Related papers  (2020-08-13T10:45:31Z)
• Revisiting Parameter Sharing in Multi-Agent Deep Reinforcement Learning [14.017603575774361]
  We formalize the notion of agent indication and prove that it enables convergence to optimal policies for the first time. Next, we formally introduce methods to extend parameter sharing to learning in heterogeneous observation and action spaces.
  arXiv  Detail & Related papers  (2020-05-27T20:14:28Z)
• Meta Reinforcement Learning with Autonomous Inference of Subtask Dependencies [57.27944046925876]
  We propose and address a novel few-shot RL problem, where a task is characterized by a subtask graph. Instead of directly learning a meta-policy, we develop a Meta-learner with Subtask Graph Inference. Our experiment results on two grid-world domains and StarCraft II environments show that the proposed method is able to accurately infer the latent task parameter.
  arXiv  Detail & Related papers  (2020-01-01T17:34:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.