Heterogeneity in Multi-Agent Reinforcement Learning

• URL: http://arxiv.org/abs/2512.22941v1
• Date: Sun, 28 Dec 2025 14:07:31 GMT
• Title: Heterogeneity in Multi-Agent Reinforcement Learning
• Authors: Tianyi Hu, Zhiqiang Pu, Yuan Wang, Tenghai Qiu, Min Chen, Xin Yu,
• Abstract summary: Heterogeneity is a fundamental property in multi-agent reinforcement learning (MARL)<n>This paper systematically discusses Heterogeneity in MARL from the perspectives of definition, quantification, and utilization.
• Score: 18.710455308404452
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Heterogeneity is a fundamental property in multi-agent reinforcement learning (MARL), which is closely related not only to the functional differences of agents, but also to policy diversity and environmental interactions. However, the MARL field currently lacks a rigorous definition and deeper understanding of heterogeneity. This paper systematically discusses heterogeneity in MARL from the perspectives of definition, quantification, and utilization. First, based on an agent-level modeling of MARL, we categorize heterogeneity into five types and provide mathematical definitions. Second, we define the concept of heterogeneity distance and propose a practical quantification method. Third, we design a heterogeneity-based multi-agent dynamic parameter sharing algorithm as an example of the application of our methodology. Case studies demonstrate that our method can effectively identify and quantify various types of agent heterogeneity. Experimental results show that the proposed algorithm, compared to other parameter sharing baselines, has better interpretability and stronger adaptability. The proposed methodology will help the MARL community gain a more comprehensive and profound understanding of heterogeneity, and further promote the development of practical algorithms.

Related papers

• MARTI-MARS$^2$: Scaling Multi-Agent Self-Search via Reinforcement Learning for Code Generation [64.2621682259008]
  Multi-Agent Reinforced Training and Inference Framework with Self-Search Scaling (MARTI-MARS2)<n>We propose a Multi-Agent Reinforced Training and Inference Framework with Self-Search Scaling (MARTI-MARS2) to integrate policy learning with multi-agent tree search.<n>We show that MARTI-MARS2 achieves 77.7%, outperforming strong baselines like GPT-5.1 on challenging code generation benchmarks.
  arXiv  Detail & Related papers  (2026-02-08T07:28:44Z)
• Understanding Agent Scaling in LLM-Based Multi-Agent Systems via Diversity [48.409368133277525]
  We present an information-theoretic framework showing that MAS performance is bounded by the intrinsic task uncertainty, not by agent count.<n>We derive architecture-agnostic bounds demonstrating that improvements depend on how many effective channels the system accesses.<n>Our results provide principled guidelines for building efficient and robust MAS through diversity-aware design.
  arXiv  Detail & Related papers  (2026-02-03T17:58:10Z)
• HyperMARL: Adaptive Hypernetworks for Multi-Agent RL [13.029350832809582]
  Multi-agent reinforcement learning (MARL) requires policies to express homogeneous, specialised, or mixed behaviours.<n>We propose a solution built on a key insight: an agent-conditioned hypernetwork can generate agent-specific parameters and decouple observation- and agent-conditioned gradients.<n>Our resulting method, HyperMARL, avoids the complexities of prior work and empirically reduces policy gradient variance.
  arXiv  Detail & Related papers  (2024-12-05T15:09:51Z)
• Improving Global Parameter-sharing in Physically Heterogeneous Multi-agent Reinforcement Learning with Unified Action Space [22.535906675532196]
  In a multi-agent system, action semantics indicates the different influences of agents' actions toward other entities. Previous multi-agent reinforcement learning (MARL) algorithms apply global parameter-sharing across different types of heterogeneous agents. We introduce the Unified Action Space (UAS) to fulfill the requirement.
  arXiv  Detail & Related papers  (2024-08-14T09:15:11Z)
• POGEMA: A Benchmark Platform for Cooperative Multi-Agent Pathfinding [76.67608003501479]
  We introduce POGEMA, a comprehensive set of tools that includes a fast environment for learning, a problem instance generator, and a visualization toolkit.<n>We also introduce and define an evaluation protocol that specifies a range of domain-related metrics, computed based on primary evaluation indicators.<n>The results of this comparison, which involves a variety of state-of-the-art MARL, search-based, and hybrid methods, are presented.
  arXiv  Detail & Related papers  (2024-07-20T16:37:21Z)
• Heterogeneous Multi-Agent Reinforcement Learning for Zero-Shot Scalable Collaboration [5.326588461041464]
  Multi-agent reinforcement learning (MARL) is transforming fields like autonomous vehicle networks. MARL strategies for different roles can be updated flexibly according to the scales, which is still a challenge for current MARL frameworks. We propose a novel MARL framework named Scalable and Heterogeneous Proximal Policy Optimization (SHPPO) We show SHPPO exhibits superior performance in classic MARL environments like Starcraft Multi-Agent Challenge (SMAC) and Google Research Football (GRF)
  arXiv  Detail & Related papers  (2024-04-05T03:02:57Z)
• Enhancing Representations through Heterogeneous Self-Supervised Learning [63.453024759497545]
  We propose Heterogeneous Self-Supervised Learning (HSSL), which enforces a base model to learn from an auxiliary head whose architecture is heterogeneous from the base model.<n>The HSSL endows the base model with new characteristics in a representation learning way without structural changes.<n>The HSSL is compatible with various self-supervised methods, achieving superior performances on various downstream tasks.
  arXiv  Detail & Related papers  (2023-10-08T10:44:05Z)
• Heterogeneous Multi-Robot Reinforcement Learning [7.22614468437919]
  Heterogeneous Graph Neural Network Proximal Policy Optimization is a paradigm for training heterogeneous MARL policies. We present a characterization of techniques that homogeneous models can leverage to emulate heterogeneous behavior.
  arXiv  Detail & Related papers  (2023-01-17T19:05:17Z)
• Highlights of Semantics in Multi-objective Genetic Programming [0.0]
  This research expands upon the current understanding of semantics in Genetic programming (GP) by proposing a new approach: Semantic-based Distance as an additional criteriOn (SDO) Our work included an expansive analysis of the GP in terms of performance and diversity metrics, using two additional semantic-based approaches, namely Semantic Similarity-based Crossover ( SCC) and Semantic-based Crowding Distance (SCD) Using highly-unbalanced binary classification datasets, we demonstrated that the newly proposed approach of SDO consistently generated more non-dominated solutions, with better diversity and improved hypervolume results.
  arXiv  Detail & Related papers  (2022-06-10T11:49:45Z)
• Consistency and Diversity induced Human Motion Segmentation [231.36289425663702]
  We propose a novel Consistency and Diversity induced human Motion (CDMS) algorithm. Our model factorizes the source and target data into distinct multi-layer feature spaces. A multi-mutual learning strategy is carried out to reduce the domain gap between the source and target data.
  arXiv  Detail & Related papers  (2022-02-10T06:23:56Z)
• Multivariate feature ranking of gene expression data [62.997667081978825]
  We propose two new multivariate feature ranking methods based on pairwise correlation and pairwise consistency. We statistically prove that the proposed methods outperform the state of the art feature ranking methods Clustering Variation, Chi Squared, Correlation, Information Gain, ReliefF and Significance.
  arXiv  Detail & Related papers  (2021-11-03T17:19:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.