Contrasting Centralized and Decentralized Critics in Multi-Agent Reinforcement Learning

• URL: http://arxiv.org/abs/2102.04402v1
• Date: Mon, 8 Feb 2021 18:08:11 GMT
• Title: Contrasting Centralized and Decentralized Critics in Multi-Agent Reinforcement Learning
• Authors: Xueguang Lyu, Yuchen Xiao, Brett Daley, Christopher Amato
• Abstract summary: Training for Decentralized Execution, where agents are trained offline using centralized information but execute in a decentralized manner online, has gained popularity in the multi-agent reinforcement learning community. In particular, actor-critic methods with a centralized critic and decentralized actors are a common instance of this idea. We analyze centralized and decentralized critic approaches, providing a deeper understanding of the implications of critic choice.
• Score: 19.66161324837036
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Centralized Training for Decentralized Execution, where agents are trained offline using centralized information but execute in a decentralized manner online, has gained popularity in the multi-agent reinforcement learning community. In particular, actor-critic methods with a centralized critic and decentralized actors are a common instance of this idea. However, the implications of using a centralized critic in this context are not fully discussed and understood even though it is the standard choice of many algorithms. We therefore formally analyze centralized and decentralized critic approaches, providing a deeper understanding of the implications of critic choice. Because our theory makes unrealistic assumptions, we also empirically compare the centralized and decentralized critic methods over a wide set of environments to validate our theories and to provide practical advice. We show that there exist misconceptions regarding centralized critics in the current literature and show that the centralized critic design is not strictly beneficial, but rather both centralized and decentralized critics have different pros and cons that should be taken into account by algorithm designers.

Related papers

• On Centralized Critics in Multi-Agent Reinforcement Learning [16.361249170514828]
  Training for Decentralized Execution has become a popular approach in Multi-Agent Reinforcement Learning. We analyze the effect of using state-based critics in partially observable environments.
  arXiv  Detail & Related papers  (2024-08-26T19:27:06Z)
• Decentralized Social Networks and the Future of Free Speech Online [4.249974621573213]
  Decentralized social networks like Mastodon and BlueSky are trending topics that have drawn much attention and discussion in recent years. This article critically assesses the decentralization project's prospect for communications online. It uses normative theories of free speech to examine whether and how the decentralization design could facilitate users' freedom of expression online.
  arXiv  Detail & Related papers  (2024-06-11T04:18:53Z)
• Exploring the Robustness of Decentralized Training for Large Language Models [51.41850749014054]
  Decentralized training of large language models has emerged as an effective way to democratize this technology. This paper explores the robustness of decentralized training from three main perspectives.
  arXiv  Detail & Related papers  (2023-12-01T04:04:03Z)
• Communication-Efficient Actor-Critic Methods for Homogeneous Markov Games [6.589813623221242]
  Policy sharing is crucial to efficient learning in certain tasks yet lacks theoretical justification. We develop the first consensus-based decentralized actor-critic method. We also develop practical algorithms based on our decentralized actor-critic method to reduce the communication cost during training.
  arXiv  Detail & Related papers  (2022-02-18T20:35:00Z)
• Byzantine-Robust Decentralized Learning via ClippedGossip [61.03711813598128]
  We propose a ClippedGossip algorithm for Byzantine-robust consensus optimization. We demonstrate the encouraging empirical performance of ClippedGossip under a large number of attacks.
  arXiv  Detail & Related papers  (2022-02-03T12:04:36Z)
• A Deeper Understanding of State-Based Critics in Multi-Agent Reinforcement Learning [17.36759906285316]
  We show that state-based critics can introduce bias in the policy estimates, potentially undermining the guarantees of the algorithm. We also show that, even if the state-based critics do not introduce any bias, they can still result in a larger gradient variance, contrary to the common intuition.
  arXiv  Detail & Related papers  (2022-01-03T14:51:30Z)
• Consensus Control for Decentralized Deep Learning [72.50487751271069]
  Decentralized training of deep learning models enables on-device learning over networks, as well as efficient scaling to large compute clusters. We show in theory that when the training consensus distance is lower than a critical quantity, decentralized training converges as fast as the centralized counterpart. Our empirical insights allow the principled design of better decentralized training schemes that mitigate the performance drop.
  arXiv  Detail & Related papers  (2021-02-09T13:58:33Z)
• Taxonomy of Centralization in Public Blockchain Systems: A Systematic Literature Review [2.1315215140430683]
  Bitcoin introduced delegation of control over a monetary system from a select few to all who participate in that system. This delegation is known as the decentralization of controlling power and is a powerful security mechanism for the ecosystem. Recent studies have observed a trend of increased centralization in cryptocurrencies such as Bitcoin and Governance.
  arXiv  Detail & Related papers  (2020-09-26T08:58:48Z)
• Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Critics [65.51757376525798]
  Overestimation bias is one of the major impediments to accurate off-policy learning. This paper investigates a novel way to alleviate the overestimation bias in a continuous control setting. Our method---Truncated Quantile Critics, TQC,---blends three ideas: distributional representation of a critic, truncation of critics prediction, and ensembling of multiple critics.
  arXiv  Detail & Related papers  (2020-05-08T19:52:26Z)
• F2A2: Flexible Fully-decentralized Approximate Actor-critic for Cooperative Multi-agent Reinforcement Learning [110.35516334788687]
  Decentralized multi-agent reinforcement learning algorithms are sometimes unpractical in complicated applications. We propose a flexible fully decentralized actor-critic MARL framework, which can handle large-scale general cooperative multi-agent setting. Our framework can achieve scalability and stability for large-scale environment and reduce information transmission.
  arXiv  Detail & Related papers  (2020-04-17T14:56:29Z)
• Byzantine-resilient Decentralized Stochastic Gradient Descent [85.15773446094576]
  We present an in-depth study towards the Byzantine resilience of decentralized learning systems. We propose UBAR, a novel algorithm to enhance decentralized learning with Byzantine Fault Tolerance.
  arXiv  Detail & Related papers  (2020-02-20T05:11:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.