Heterogeneous Multi-Robot Reinforcement Learning

• URL: http://arxiv.org/abs/2301.07137v1
• Date: Tue, 17 Jan 2023 19:05:17 GMT
• Title: Heterogeneous Multi-Robot Reinforcement Learning
• Authors: Matteo Bettini, Ajay Shankar, Amanda Prorok
• Abstract summary: Heterogeneous Graph Neural Network Proximal Policy Optimization is a paradigm for training heterogeneous MARL policies. We present a characterization of techniques that homogeneous models can leverage to emulate heterogeneous behavior.
• Score: 7.22614468437919
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Cooperative multi-robot tasks can benefit from heterogeneity in the robots' physical and behavioral traits. In spite of this, traditional Multi-Agent Reinforcement Learning (MARL) frameworks lack the ability to explicitly accommodate policy heterogeneity, and typically constrain agents to share neural network parameters. This enforced homogeneity limits application in cases where the tasks benefit from heterogeneous behaviors. In this paper, we crystallize the role of heterogeneity in MARL policies. Towards this end, we introduce Heterogeneous Graph Neural Network Proximal Policy Optimization (HetGPPO), a paradigm for training heterogeneous MARL policies that leverages a Graph Neural Network for differentiable inter-agent communication. HetGPPO allows communicating agents to learn heterogeneous behaviors while enabling fully decentralized training in partially observable environments. We complement this with a taxonomical overview that exposes more heterogeneity classes than previously identified. To motivate the need for our model, we present a characterization of techniques that homogeneous models can leverage to emulate heterogeneous behavior, and show how this "apparent heterogeneity" is brittle in real-world conditions. Through simulations and real-world experiments, we show that: (i) when homogeneous methods fail due to strong heterogeneous requirements, HetGPPO succeeds, and, (ii) when homogeneous methods are able to learn apparently heterogeneous behaviors, HetGPPO achieves higher resilience to both training and deployment noise.

Related papers

• Unified Generative and Discriminative Training for Multi-modal Large Language Models [88.84491005030316]
  Generative training has enabled Vision-Language Models (VLMs) to tackle various complex tasks. Discriminative training, exemplified by models like CLIP, excels in zero-shot image-text classification and retrieval. This paper proposes a unified approach that integrates the strengths of both paradigms.
  arXiv  Detail & Related papers  (2024-11-01T01:51:31Z)
• Interpreting artificial neural networks to detect genome-wide association signals for complex traits [0.0]
  Investigating the genetic architecture of complex diseases is challenging due to the highly polygenic and interactive landscape of genetic and environmental factors. We trained artificial neural networks for predicting complex traits using both simulated and real genotype/phenotype datasets.
  arXiv  Detail & Related papers  (2024-07-26T15:20:42Z)
• Semantically Rich Local Dataset Generation for Explainable AI in Genomics [0.716879432974126]
  Black box deep learning models trained on genomic sequences excel at predicting the outcomes of different gene regulatory mechanisms. We propose using Genetic Programming to generate datasets by evolving perturbations in sequences that contribute to their semantic diversity.
  arXiv  Detail & Related papers  (2024-07-03T10:31:30Z)
• Task Groupings Regularization: Data-Free Meta-Learning with Heterogeneous Pre-trained Models [83.02797560769285]
  Data-Free Meta-Learning (DFML) aims to derive knowledge from a collection of pre-trained models without accessing their original data. Current methods often overlook the heterogeneity among pre-trained models, which leads to performance degradation due to task conflicts. We propose Task Groupings Regularization, a novel approach that benefits from model heterogeneity by grouping and aligning conflicting tasks.
  arXiv  Detail & Related papers  (2024-05-26T13:11:55Z)
• Heterogeneous Multi-Agent Reinforcement Learning for Zero-Shot Scalable Collaboration [5.326588461041464]
  Multi-agent reinforcement learning (MARL) is transforming fields like autonomous vehicle networks. MARL strategies for different roles can be updated flexibly according to the scales, which is still a challenge for current MARL frameworks. We propose a novel MARL framework named Scalable and Heterogeneous Proximal Policy Optimization (SHPPO) We show SHPPO exhibits superior performance in classic MARL environments like Starcraft Multi-Agent Challenge (SMAC) and Google Research Football (GRF)
  arXiv  Detail & Related papers  (2024-04-05T03:02:57Z)
• System Neural Diversity: Measuring Behavioral Heterogeneity in Multi-Agent Learning [8.280943341629161]
  We introduce System Neural Diversity (SND): a measure of behavioral heterogeneity in multi-agent systems. We show how SND allows us to measure latent resilience skills acquired by the agents, while other proxies, such as task performance (reward), fail. We demonstrate how this paradigm can be used to bootstrap the exploration phase, finding optimal policies faster.
  arXiv  Detail & Related papers  (2023-05-03T13:58:13Z)
• Differentiable Agent-based Epidemiology [71.81552021144589]
  We introduce GradABM: a scalable, differentiable design for agent-based modeling that is amenable to gradient-based learning with automatic differentiation. GradABM can quickly simulate million-size populations in few seconds on commodity hardware, integrate with deep neural networks and ingest heterogeneous data sources.
  arXiv  Detail & Related papers  (2022-07-20T07:32:02Z)
• Efficient Model-based Multi-agent Reinforcement Learning via Optimistic Equilibrium Computation [93.52573037053449]
  H-MARL (Hallucinated Multi-Agent Reinforcement Learning) learns successful equilibrium policies after a few interactions with the environment. We demonstrate our approach experimentally on an autonomous driving simulation benchmark.
  arXiv  Detail & Related papers  (2022-03-14T17:24:03Z)
• Automatic Generation of Individual Fuzzy Cognitive Maps from Longitudinal Data [0.0]
  Fuzzy Cognitive Maps (FCMs) are computational models that represent how factors (nodes) change over discrete interactions. In this paper, we use Genetic Algorithms to create one FCM for each agent, thus providing the means to automatically create a virtual population with heterogeneous behaviors.
  arXiv  Detail & Related papers  (2022-02-14T22:11:58Z)
• Cooperative Policy Learning with Pre-trained Heterogeneous Observation Representations [51.8796674904734]
  We propose a new cooperative learning framework with pre-trained heterogeneous observation representations. We employ an encoder-decoder based graph attention to learn the intricate interactions and heterogeneous representations.
  arXiv  Detail & Related papers  (2020-12-24T04:52:29Z)
• A Multi-Semantic Metapath Model for Large Scale Heterogeneous Network Representation Learning [52.83948119677194]
  We propose a multi-semantic metapath (MSM) model for large scale heterogeneous representation learning. Specifically, we generate multi-semantic metapath-based random walks to construct the heterogeneous neighborhood to handle the unbalanced distributions. We conduct systematical evaluations for the proposed framework on two challenging datasets: Amazon and Alibaba.
  arXiv  Detail & Related papers  (2020-07-19T22:50:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.