Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning

• URL: http://arxiv.org/abs/2003.08839v2
• Date: Thu, 27 Aug 2020 13:45:29 GMT
• Title: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning
• Authors: Tabish Rashid, Mikayel Samvelyan, Christian Schroeder de Witt, Gregory Farquhar, Jakob Foerster, Shimon Whiteson
• Abstract summary: QMIX is a novel value-based method that can train decentralised policies in a centralised end-to-end fashion. We propose the StarCraft Multi-Agent Challenge (SMAC) as a new benchmark for deep multi-agent reinforcement learning.
• Score: 55.20040781688844
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In many real-world settings, a team of agents must coordinate its behaviour while acting in a decentralised fashion. At the same time, it is often possible to train the agents in a centralised fashion where global state information is available and communication constraints are lifted. Learning joint action-values conditioned on extra state information is an attractive way to exploit centralised learning, but the best strategy for then extracting decentralised policies is unclear. Our solution is QMIX, a novel value-based method that can train decentralised policies in a centralised end-to-end fashion. QMIX employs a mixing network that estimates joint action-values as a monotonic combination of per-agent values. We structurally enforce that the joint-action value is monotonic in the per-agent values, through the use of non-negative weights in the mixing network, which guarantees consistency between the centralised and decentralised policies. To evaluate the performance of QMIX, we propose the StarCraft Multi-Agent Challenge (SMAC) as a new benchmark for deep multi-agent reinforcement learning. We evaluate QMIX on a challenging set of SMAC scenarios and show that it significantly outperforms existing multi-agent reinforcement learning methods.

Related papers

• Soft-QMIX: Integrating Maximum Entropy For Monotonic Value Function Factorization [5.54284350152423]
  We propose an enhancement to QMIX by incorporating an additional local Qvalue learning method within the maximum entropy RL framework. Our approach constrains the local Q-value estimates to maintain the correct ordering of all actions. We theoretically prove the monotonic improvement and convergence of our method to an optimal solution.
  arXiv  Detail & Related papers  (2024-06-20T01:55:08Z)
• Expeditious Saliency-guided Mix-up through Random Gradient Thresholding [89.59134648542042]
  Mix-up training approaches have proven to be effective in improving the generalization ability of Deep Neural Networks. In this paper, inspired by the superior qualities of each direction over one another, we introduce a novel method that lies at the junction of the two routes. We name our method R-Mix following the concept of "Random Mix-up" In order to address the question of whether there exists a better decision protocol, we train a Reinforcement Learning agent that decides the mix-up policies.
  arXiv  Detail & Related papers  (2022-12-09T14:29:57Z)
• Locality Matters: A Scalable Value Decomposition Approach for Cooperative Multi-Agent Reinforcement Learning [52.7873574425376]
  Cooperative multi-agent reinforcement learning (MARL) faces significant scalability issues due to state and action spaces that are exponentially large in the number of agents. We propose a novel, value-based multi-agent algorithm called LOMAQ, which incorporates local rewards in the Training Decentralized Execution paradigm.
  arXiv  Detail & Related papers  (2021-09-22T10:08:15Z)
• MMD-MIX: Value Function Factorisation with Maximum Mean Discrepancy for Cooperative Multi-Agent Reinforcement Learning [15.972363414919279]
  MMD-mix is a method that combines distributional reinforcement learning and value decomposition. The experiments demonstrate that MMD-mix outperforms prior baselines in the Star Multi-Agent Challenge (SMAC) environment.
  arXiv  Detail & Related papers  (2021-06-22T10:21:00Z)
• Credit Assignment with Meta-Policy Gradient for Multi-Agent Reinforcement Learning [29.895142928565228]
  We propose a general meta-learning-based Mixing Network with Meta Policy Gradient(MNMPG) framework to distill the global hierarchy for delicate reward decomposition. Experiments on the StarCraft II micromanagement benchmark demonstrate that our method just with a simple utility network is able to outperform the current state-of-the-art MARL algorithms.
  arXiv  Detail & Related papers  (2021-02-24T12:03:37Z)
• Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge? [100.48692829396778]
  Independent PPO (IPPO) is a form of independent learning in which each agent simply estimates its local value function. IPPO's strong performance may be due to its robustness to some forms of environment non-stationarity.
  arXiv  Detail & Related papers  (2020-11-18T20:29:59Z)
• F2A2: Flexible Fully-decentralized Approximate Actor-critic for Cooperative Multi-agent Reinforcement Learning [110.35516334788687]
  Decentralized multi-agent reinforcement learning algorithms are sometimes unpractical in complicated applications. We propose a flexible fully decentralized actor-critic MARL framework, which can handle large-scale general cooperative multi-agent setting. Our framework can achieve scalability and stability for large-scale environment and reduce information transmission.
  arXiv  Detail & Related papers  (2020-04-17T14:56:29Z)
• FACMAC: Factored Multi-Agent Centralised Policy Gradients [103.30380537282517]
  We propose FACtored Multi-Agent Centralised policy gradients (FACMAC) It is a new method for cooperative multi-agent reinforcement learning in both discrete and continuous action spaces. We evaluate FACMAC on variants of the multi-agent particle environments, a novel multi-agent MuJoCo benchmark, and a challenging set of StarCraft II micromanagement tasks.
  arXiv  Detail & Related papers  (2020-03-14T21:29:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.