Hierarchical Message-Passing Policies for Multi-Agent Reinforcement Learning

• URL: http://arxiv.org/abs/2507.23604v1
• Date: Thu, 31 Jul 2025 14:42:12 GMT
• Title: Hierarchical Message-Passing Policies for Multi-Agent Reinforcement Learning
• Authors: Tommaso Marzi, Cesare Alippi, Andrea Cini,
• Abstract summary: We propose a novel and effective methodology for learning multi-agent hierarchies of message-passing policies.<n>Agents at lower levels in the hierarchy receive goals from the upper levels and exchange messages with neighboring agents at the same level.<n>Results on relevant benchmarks show that our method performs favorably compared to the state of the art.
• Score: 19.739901034066587
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Decentralized Multi-Agent Reinforcement Learning (MARL) methods allow for learning scalable multi-agent policies, but suffer from partial observability and induced non-stationarity. These challenges can be addressed by introducing mechanisms that facilitate coordination and high-level planning. Specifically, coordination and temporal abstraction can be achieved through communication (e.g., message passing) and Hierarchical Reinforcement Learning (HRL) approaches to decision-making. However, optimization issues limit the applicability of hierarchical policies to multi-agent systems. As such, the combination of these approaches has not been fully explored. To fill this void, we propose a novel and effective methodology for learning multi-agent hierarchies of message-passing policies. We adopt the feudal HRL framework and rely on a hierarchical graph structure for planning and coordination among agents. Agents at lower levels in the hierarchy receive goals from the upper levels and exchange messages with neighboring agents at the same level. To learn hierarchical multi-agent policies, we design a novel reward-assignment method based on training the lower-level policies to maximize the advantage function associated with the upper levels. Results on relevant benchmarks show that our method performs favorably compared to the state of the art.

Related papers

• AnyMAC: Cascading Flexible Multi-Agent Collaboration via Next-Agent Prediction [70.60422261117816]
  We propose a new framework that rethinks multi-agent coordination through a sequential structure rather than a graph structure.<n>Our method focuses on two key directions: (1) Next-Agent Prediction, which selects the most suitable agent role at each step, and (2) Next-Context Selection, which enables each agent to selectively access relevant information from any previous step.
  arXiv  Detail & Related papers  (2025-06-21T18:34:43Z)
• MultiAgentBench: Evaluating the Collaboration and Competition of LLM agents [59.825725526176655]
  Large Language Models (LLMs) have shown remarkable capabilities as autonomous agents.<n>Existing benchmarks either focus on single-agent tasks or are confined to narrow domains, failing to capture the dynamics of multi-agent coordination and competition.<n>We introduce MultiAgentBench, a benchmark designed to evaluate LLM-based multi-agent systems across diverse, interactive scenarios.
  arXiv  Detail & Related papers  (2025-03-03T05:18:50Z)
• TAG: A Decentralized Framework for Multi-Agent Hierarchical Reinforcement Learning [4.591755344464076]
  We introduce TAME Agent Framework (TAG), a framework for constructing fully decentralized hierarchical multi-agent systems.<n>TAG standardizes information flow between levels while preserving loose coupling, allowing for seamless integration of diverse agent types.<n>Our results show that decentralized hierarchical organization enhances both learning speed and final performance, positioning TAG as a promising direction for scalable multi-agent systems.
  arXiv  Detail & Related papers  (2025-02-21T12:52:16Z)
• Hierarchical Reinforcement Learning for Optimal Agent Grouping in Cooperative Systems [0.4759142872591625]
  This paper presents a hierarchical reinforcement learning (RL) approach to address the agent grouping or pairing problem in cooperative multi-agent systems.<n>By employing a hierarchical RL framework, we distinguish between high-level decisions of grouping and low-level agents' actions.<n>We incorporate permutation-in neural networks to handle the homogeneity and cooperation among agents, enabling effective coordination.
  arXiv  Detail & Related papers  (2025-01-11T14:22:10Z)
• From Novice to Expert: LLM Agent Policy Optimization via Step-wise Reinforcement Learning [62.54484062185869]
  We introduce StepAgent, which utilizes step-wise reward to optimize the agent's reinforcement learning process.<n>We propose implicit-reward and inverse reinforcement learning techniques to facilitate agent reflection and policy adjustment.
  arXiv  Detail & Related papers  (2024-11-06T10:35:11Z)
• Hierarchical Reinforcement Learning for Temporal Abstraction of Listwise Recommendation [51.06031200728449]
  We propose a novel framework called mccHRL to provide different levels of temporal abstraction on listwise recommendation.<n>Within the hierarchical framework, the high-level agent studies the evolution of user perception, while the low-level agent produces the item selection policy.<n>Results observe significant performance improvement by our method, compared with several well-known baselines.
  arXiv  Detail & Related papers  (2024-09-11T17:01:06Z)
• A Provably Efficient Option-Based Algorithm for both High-Level and Low-Level Learning [54.20447310988282]
  We present a meta-algorithm alternating between regret minimization algorithms instanced at different (high and low) temporal abstractions.<n>At the higher level, we treat the problem as a Semi-Markov Decision Process (SMDP), with fixed low-level policies, while at a lower level, inner option policies are learned with a fixed high-level policy.
  arXiv  Detail & Related papers  (2024-06-21T13:17:33Z)
• Efficient Adaptation in Mixed-Motive Environments via Hierarchical Opponent Modeling and Planning [51.52387511006586]
  We propose Hierarchical Opponent modeling and Planning (HOP), a novel multi-agent decision-making algorithm. HOP is hierarchically composed of two modules: an opponent modeling module that infers others' goals and learns corresponding goal-conditioned policies. HOP exhibits superior few-shot adaptation capabilities when interacting with various unseen agents, and excels in self-play scenarios.
  arXiv  Detail & Related papers  (2024-06-12T08:48:06Z)
• Feudal Graph Reinforcement Learning [18.069747511100132]
  Graph-based representations and message-passing modular policies constitute prominent approaches to tackling composable control problems in reinforcement learning (RL)<n>We propose a novel methodology, named Feudal Graph Reinforcement Learning (FGRL), that addresses such challenges by relying on hierarchical RL and a pyramidal message-passing architecture.<n>In particular, FGRL defines a hierarchy of policies where high-level commands are propagated from the top of the hierarchy down through a layered graph structure.
  arXiv  Detail & Related papers  (2023-04-11T09:51:13Z)
• Option-Aware Adversarial Inverse Reinforcement Learning for Robotic Control [44.77500987121531]
  Hierarchical Imitation Learning (HIL) has been proposed to recover highly-complex behaviors in long-horizon tasks from expert demonstrations. We develop a novel HIL algorithm based on Adversarial Inverse Reinforcement Learning. We also propose a Variational Autoencoder framework for learning with our objectives in an end-to-end fashion.
  arXiv  Detail & Related papers  (2022-10-05T00:28:26Z)
• Hierarchical Reinforcement Learning with Opponent Modeling for Distributed Multi-agent Cooperation [13.670618752160594]
  Deep reinforcement learning (DRL) provides a promising approach for multi-agent cooperation through the interaction of the agents and environments. Traditional DRL solutions suffer from the high dimensions of multiple agents with continuous action space during policy search. We propose a hierarchical reinforcement learning approach with high-level decision-making and low-level individual control for efficient policy search.
  arXiv  Detail & Related papers  (2022-06-25T19:09:29Z)
• HAVEN: Hierarchical Cooperative Multi-Agent Reinforcement Learning with Dual Coordination Mechanism [17.993973801986677]
  Multi-agent reinforcement learning often suffers from the exponentially larger action space caused by a large number of agents. We propose a novel value decomposition framework HAVEN based on hierarchical reinforcement learning for the fully cooperative multi-agent problems.
  arXiv  Detail & Related papers  (2021-10-14T10:43:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.