Scalable Neighborhood-Based Multi-Agent Actor-Critic

• URL: http://arxiv.org/abs/2604.18190v1
• Date: Mon, 20 Apr 2026 12:45:59 GMT
• Title: Scalable Neighborhood-Based Multi-Agent Actor-Critic
• Abstract summary: We propose MADDPG-K, a scalable extension to Multi-Agent Deep Deterministic Policy Gradient (MADDPG)<n>MADDPG-K mitigates each agent's critic to the $$ closest agents under a chosen metric which in our case is Euclidean distance.<n>We validate our method empirically across cooperative and adversarial environments from the Multi-Particle Environment suite.
• Score: 3.007949058551534
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We propose MADDPG-K, a scalable extension to Multi-Agent Deep Deterministic Policy Gradient (MADDPG) that addresses the computational limitations of centralized critic approaches. Centralized critics, which condition on the observations and actions of all agents, have demonstrated significant performance gains in cooperative and competitive multi-agent settings. However, their critic networks grow linearly in input size with the number of agents, making them increasingly expensive to train at scale. MADDPG-K mitigates this by restricting each agent's critic to the $k$ closest agents under a chosen metric which in our case is Euclidean distance. This ensures a constant-size critic input regardless of the total agent count. We analyze the complexity of this approach, showing that the quadratic cost it retains arises from cheap scalar distance computations rather than the expensive neural network matrix multiplications that bottleneck standard MADDPG. We validate our method empirically across cooperative and adversarial environments from the Multi-Particle Environment suite, demonstrating competitive or superior performance compared to MADDPG, faster convergence in cooperative settings, and better runtime scaling as the number of agents grows. Our code is available at https://github.com/TimGop/MADDPG-K .

Related papers

• Multi-Agent Off-Policy Deep Reinforcement Learning for Smart Campus Coverage [0.6999740786886536]
  Deep reinforcement learning (DRL) has recently gained a great attention to its real-time adaptation and effectiveness in complex optimization problems.<n>This paper investigates efficient computational users in a dense multi-agent framework.
  arXiv  Detail & Related papers  (2026-08-19T15:41:32Z)
• Comparative Study of Multi-Agent Actor-Critic Algorithms in Parameterized Action Reinforcement Learning [0.0]
  Multi-Agent Greedy Actor-Critic (MAGAC), Multi-Agent Soft Actor-Critic (MASAC), and Multi-Agent Truncated Quantile Critics (MATQC)<n>We evaluate the algorithms on the Platform-v0 and Goal-v0 benchmarks against their single-agent counterparts.<n>Results show that the multi-agent framework consistently improves Greedy Actor-Critic performance.
  arXiv  Detail & Related papers  (2026-07-21T14:03:29Z)
• Agent Q-Mix: Selecting the Right Action for LLM Multi-Agent Systems through Reinforcement Learning [68.85408801740228]
  We propose textbfAgent Q-Mix, a reinforcement learning framework that reformulates topology selection as a cooperative Multi-Agent Reinforcement Learning problem.<n>Our method learns decentralized communication decisions using Qmix value factorization, where each agent selects from a set of communication actions that jointly induce a round-wise communication graph.<n>Agent Q-Mix achieves the highest average accuracy compared to existing methods while demonstrating superior token efficiency and robustness against agent failure.
  arXiv  Detail & Related papers  (2026-04-01T00:38:24Z)
• Near-Optimal Online Learning for Multi-Agent Submodular Coordination: Tight Approximation and Communication Efficiency [52.60557300927007]
  We present a $textbfMA-OSMA$ algorithm to transfer the discrete submodular problem into a continuous optimization.<n>We also introduce a projection-free $textbfMA-OSEA$ algorithm, which effectively utilizes the KL divergence by mixing a uniform distribution.<n>Our algorithms significantly improve the $(frac11+c)$-approximation provided by the state-of-the-art OSG algorithm.
  arXiv  Detail & Related papers  (2025-02-07T15:57:56Z)
• Conservative DDPG -- Pessimistic RL without Ensemble [48.61228614796803]
  DDPG is hindered by the overestimation bias problem. Traditional solutions to this bias involve ensemble-based methods. We propose a straightforward solution using a $Q$-target and incorporating a behavioral cloning (BC) loss penalty.
  arXiv  Detail & Related papers  (2024-03-08T23:59:38Z)
• Revisiting the Gumbel-Softmax in MADDPG [11.292086312664383]
  DDPG is an algorithm designed for continuous action spaces, where the gradient of the state-action value function exists. For this algorithm to work in discrete action spaces, discrete gradient estimation must be performed. This paper explores several alternatives and integrates them into MADDPG for discrete grid-world scenarios.
  arXiv  Detail & Related papers  (2023-02-23T06:13:51Z)
• Byzantine-Robust Online and Offline Distributed Reinforcement Learning [60.970950468309056]
  We consider a distributed reinforcement learning setting where multiple agents explore the environment and communicate their experiences through a central server. $alpha$-fraction of agents are adversarial and can report arbitrary fake information. We seek to identify a near-optimal policy for the underlying Markov decision process in the presence of these adversarial agents.
  arXiv  Detail & Related papers  (2022-06-01T00:44:53Z)
• Asymptotic Convergence of Deep Multi-Agent Actor-Critic Algorithms [0.6961253535504979]
  We present sufficient conditions that ensure convergence of the multi-agent Deep Deterministic Policy Gradient (DDPG) algorithm. It is an example of one of the most popular paradigms of Deep Reinforcement Learning (DeepRL) for tackling continuous action spaces: the actor-critic paradigm.
  arXiv  Detail & Related papers  (2022-01-03T10:33:52Z)
• Off-Policy Multi-Agent Decomposed Policy Gradients [30.389041305278045]
  We investigate causes that hinder the performance of MAPG algorithms and present a multi-agent decomposed policy gradient method (DOP) DOP supports efficient off-policy learning and addresses the issue of centralized-decentralized mismatch and credit assignment. In addition, empirical evaluations on the StarCraft II micromanagement benchmark and multi-agent particle environments demonstrate that DOP significantly outperforms both state-of-the-art value-based and policy-based multi-agent reinforcement learning algorithms.
  arXiv  Detail & Related papers  (2020-07-24T02:21:55Z)
• FACMAC: Factored Multi-Agent Centralised Policy Gradients [103.30380537282517]
  We propose FACtored Multi-Agent Centralised policy gradients (FACMAC) It is a new method for cooperative multi-agent reinforcement learning in both discrete and continuous action spaces. We evaluate FACMAC on variants of the multi-agent particle environments, a novel multi-agent MuJoCo benchmark, and a challenging set of StarCraft II micromanagement tasks.
  arXiv  Detail & Related papers  (2020-03-14T21:29:09Z)
• Counterfactual Multi-Agent Policy Gradients [47.45255170608965]
  We propose a new multi-agent actor-critic method called counterfactual multi-agent (COMA) policy gradients.<n>COMA uses a centralised critic to estimate the Q-function and decentralised actors to optimise the agents' policies.<n>We evaluate COMA in the testbed of StarCraft unit micromanagement, using a decentralised variant with significant partial observability.
  arXiv  Detail & Related papers  (2017-05-24T18:52:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.