System Neural Diversity: Measuring Behavioral Heterogeneity in
Multi-Agent Learning
- URL: http://arxiv.org/abs/2305.02128v1
- Date: Wed, 3 May 2023 13:58:13 GMT
- Title: System Neural Diversity: Measuring Behavioral Heterogeneity in
Multi-Agent Learning
- Authors: Matteo Bettini, Ajay Shankar, Amanda Prorok
- Abstract summary: We introduce System Neural Diversity (SND), a measure of behavioral heterogeneity for multi-agent systems.
We show how SND constitutes an important diagnostic tool to analyze latent properties of behavioral heterogeneity.
- Score: 7.22614468437919
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Evolutionary science provides evidence that diversity confers resilience.
Yet, traditional multi-agent reinforcement learning techniques commonly enforce
homogeneity to increase training sample efficiency. When a system of learning
agents is not constrained to homogeneous policies, individual agents may
develop diverse behaviors, resulting in emergent complementarity that benefits
the system. Despite this feat, there is a surprising lack of tools that measure
behavioral diversity in systems of learning agents. Such techniques would pave
the way towards understanding the impact of diversity in collective resilience
and performance. In this paper, we introduce System Neural Diversity (SND): a
measure of behavioral heterogeneity for multi-agent systems where agents have
stochastic policies. %over a continuous state space. We discuss and prove its
theoretical properties, and compare it with alternate, state-of-the-art
behavioral diversity metrics used in cross-disciplinary domains. Through
simulations of a variety of multi-agent tasks, we show how our metric
constitutes an important diagnostic tool to analyze latent properties of
behavioral heterogeneity. By comparing SND with task reward in static tasks,
where the problem does not change during training, we show that it is key to
understanding the effectiveness of heterogeneous vs homogeneous agents. In
dynamic tasks, where the problem is affected by repeated disturbances during
training, we show that heterogeneous agents are first able to learn specialized
roles that allow them to cope with the disturbance, and then retain these roles
when the disturbance is removed. SND allows a direct measurement of this latent
resilience, while other proxies such as task performance (reward) fail to.
Related papers
- Controlling Behavioral Diversity in Multi-Agent Reinforcement Learning [8.905920197601173]
We introduce Diversity Control (DiCo), a method able to control diversity to an exact value of a given metric.
We show how DiCo can be employed as a novel paradigm to increase performance and sample efficiency in Multi-Agent Reinforcement Learning.
arXiv Detail & Related papers (2024-05-23T21:03:33Z) - Causal Graph ODE: Continuous Treatment Effect Modeling in Multi-agent
Dynamical Systems [70.84976977950075]
Real-world multi-agent systems are often dynamic and continuous, where the agents co-evolve and undergo changes in their trajectories and interactions over time.
We propose a novel model that captures the continuous interaction among agents using a Graph Neural Network (GNN) as the ODE function.
The key innovation of our model is to learn time-dependent representations of treatments and incorporate them into the ODE function, enabling precise predictions of potential outcomes.
arXiv Detail & Related papers (2024-02-29T23:07:07Z) - DCIR: Dynamic Consistency Intrinsic Reward for Multi-Agent Reinforcement
Learning [84.22561239481901]
We propose a new approach that enables agents to learn whether their behaviors should be consistent with that of other agents.
We evaluate DCIR in multiple environments including Multi-agent Particle, Google Research Football and StarCraft II Micromanagement.
arXiv Detail & Related papers (2023-12-10T06:03:57Z) - Source-free Domain Adaptation Requires Penalized Diversity [60.04618512479438]
Source-free domain adaptation (SFDA) was introduced to address knowledge transfer between different domains in the absence of source data.
In unsupervised SFDA, the diversity is limited to learning a single hypothesis on the source or learning multiple hypotheses with a shared feature extractor.
We propose a novel unsupervised SFDA algorithm that promotes representational diversity through the use of separate feature extractors.
arXiv Detail & Related papers (2023-04-06T00:20:19Z) - Heterogeneous Multi-Robot Reinforcement Learning [7.22614468437919]
Heterogeneous Graph Neural Network Proximal Policy Optimization is a paradigm for training heterogeneous MARL policies.
We present a characterization of techniques that homogeneous models can leverage to emulate heterogeneous behavior.
arXiv Detail & Related papers (2023-01-17T19:05:17Z) - Differentiable Agent-based Epidemiology [71.81552021144589]
We introduce GradABM: a scalable, differentiable design for agent-based modeling that is amenable to gradient-based learning with automatic differentiation.
GradABM can quickly simulate million-size populations in few seconds on commodity hardware, integrate with deep neural networks and ingest heterogeneous data sources.
arXiv Detail & Related papers (2022-07-20T07:32:02Z) - Policy Diagnosis via Measuring Role Diversity in Cooperative Multi-agent
RL [107.58821842920393]
We quantify the agent's behavior difference and build its relationship with the policy performance via bf Role Diversity
We find that the error bound in MARL can be decomposed into three parts that have a strong relation to the role diversity.
The decomposed factors can significantly impact policy optimization on three popular directions.
arXiv Detail & Related papers (2022-06-01T04:58:52Z) - Multi-Agent Imitation Learning with Copulas [102.27052968901894]
Multi-agent imitation learning aims to train multiple agents to perform tasks from demonstrations by learning a mapping between observations and actions.
In this paper, we propose to use copula, a powerful statistical tool for capturing dependence among random variables, to explicitly model the correlation and coordination in multi-agent systems.
Our proposed model is able to separately learn marginals that capture the local behavioral patterns of each individual agent, as well as a copula function that solely and fully captures the dependence structure among agents.
arXiv Detail & Related papers (2021-07-10T03:49:41Z) - Quantifying environment and population diversity in multi-agent
reinforcement learning [7.548322030720646]
Generalization is a major challenge for multi-agent reinforcement learning.
In this paper, we investigate and quantify the relationship between generalization and diversity in the multi-agent domain.
To better understand the effects of co-player variation, our experiments introduce a new environment-agnostic measure of behavioral diversity.
arXiv Detail & Related papers (2021-02-16T18:54:39Z) - Ubiquitous Distributed Deep Reinforcement Learning at the Edge:
Analyzing Byzantine Agents in Discrete Action Spaces [0.06554326244334865]
This paper discusses some of the challenges in multi-agent distributed deep reinforcement learning that can occur in the presence of byzantine or malfunctioning agents.
We show how wrong discrete actions can significantly affect the collaborative learning effort.
Experiments are carried out in a simulation environment using the Atari testbed for the discrete action spaces, and advantage actor-critic (A2C) for the distributed multi-agent training.
arXiv Detail & Related papers (2020-08-18T11:25:39Z) - Effective Diversity in Population Based Reinforcement Learning [38.62641968788987]
We introduce an approach to optimize all members of a population simultaneously.
Rather than using pairwise distance, we measure the volume of the entire population in a behavioral manifold.
Our algorithm Diversity via Determinants (DvD) adapts the degree of diversity during training using online learning techniques.
arXiv Detail & Related papers (2020-02-03T10:09:16Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.