Abstract: Extending transfer learning to cooperative multi-agent reinforcement learning
(MARL) has recently received much attention. In contrast to the single-agent
setting, the coordination indispensable in cooperative MARL constrains each
agent's policy. However, existing transfer methods focus exclusively on agent
policy and ignores coordination knowledge. We propose a new architecture that
realizes robust coordination knowledge transfer through appropriate
decomposition of the overall coordination into several coordination patterns.
We use a novel mixing network named level-adaptive QTransformer
(LA-QTransformer) to realize agent coordination that considers credit
assignment, with appropriate coordination patterns for different agents
realized by a novel level-adaptive Transformer (LA-Transformer) dedicated to
the transfer of coordination knowledge. In addition, we use a novel agent
network named Population Invariant agent with Transformer (PIT) to realize the
coordination transfer in more varieties of scenarios. Extensive experiments in
StarCraft II micro-management show that LA-QTransformer together with PIT
achieves superior performance compared with state-of-the-art baselines.