Multi-agent cooperation through learning-aware policy gradients

• URL: http://arxiv.org/abs/2410.18636v1
• Date: Thu, 24 Oct 2024 10:48:42 GMT
• Title: Multi-agent cooperation through learning-aware policy gradients
• Authors: Alexander Meulemans, Seijin Kobayashi, Johannes von Oswald, Nino Scherrer, Eric Elmoznino, Blake Richards, Guillaume Lajoie, Blaise Agüera y Arcas, João Sacramento,
• Abstract summary: Self-interested individuals often fail to cooperate, posing a fundamental challenge for multi-agent learning. We present the first unbiased, higher-derivative-free policy gradient algorithm for learning-aware reinforcement learning. We derive from the iterated prisoner's dilemma a novel explanation for how and when cooperation arises among self-interested learning-aware agents.
• Score: 53.63948041506278
• License:
• Abstract: Self-interested individuals often fail to cooperate, posing a fundamental challenge for multi-agent learning. How can we achieve cooperation among self-interested, independent learning agents? Promising recent work has shown that in certain tasks cooperation can be established between learning-aware agents who model the learning dynamics of each other. Here, we present the first unbiased, higher-derivative-free policy gradient algorithm for learning-aware reinforcement learning, which takes into account that other agents are themselves learning through trial and error based on multiple noisy trials. We then leverage efficient sequence models to condition behavior on long observation histories that contain traces of the learning dynamics of other agents. Training long-context policies with our algorithm leads to cooperative behavior and high returns on standard social dilemmas, including a challenging environment where temporally-extended action coordination is required. Finally, we derive from the iterated prisoner's dilemma a novel explanation for how and when cooperation arises among self-interested learning-aware agents.

Related papers

• Reciprocal Reward Influence Encourages Cooperation From Self-Interested Agents [2.1301560294088318]
  Cooperation between self-interested individuals is a widespread phenomenon in the natural world, but remains elusive in interactions between artificially intelligent agents. We introduce Reciprocators, reinforcement learning agents which are intrinsically motivated to reciprocate the influence of opponents' actions on their returns. We show that Reciprocators can be used to promote cooperation in temporally extended social dilemmas during simultaneous learning.
  arXiv  Detail & Related papers  (2024-06-03T06:07:27Z)
• Enhancing Cooperation through Selective Interaction and Long-term Experiences in Multi-Agent Reinforcement Learning [10.932974027102619]
  This study introduces a computational framework based on multi-agent reinforcement learning in the spatial Prisoner's Dilemma game. By modelling each agent using two distinct Q-networks, we disentangle the coevolutionary dynamics between cooperation and interaction.
  arXiv  Detail & Related papers  (2024-05-04T12:42:55Z)
• Decentralized and Lifelong-Adaptive Multi-Agent Collaborative Learning [57.652899266553035]
  Decentralized and lifelong-adaptive multi-agent collaborative learning aims to enhance collaboration among multiple agents without a central server. We propose DeLAMA, a decentralized multi-agent lifelong collaborative learning algorithm with dynamic collaboration graphs.
  arXiv  Detail & Related papers  (2024-03-11T09:21:11Z)
• Peer Learning: Learning Complex Policies in Groups from Scratch via Action Recommendations [16.073203911932872]
  Peer learning is a novel high-level reinforcement learning framework for agents learning in groups. We show that peer learning is able to outperform single agent learning and the baseline in several challenging OpenAI Gym domains.
  arXiv  Detail & Related papers  (2023-12-15T17:01:35Z)
• Incremental procedural and sensorimotor learning in cognitive humanoid robots [52.77024349608834]
  This work presents a cognitive agent that can learn procedures incrementally. We show the cognitive functions required in each substage and how adding new functions helps address tasks previously unsolved by the agent. Results show that this approach is capable of solving complex tasks incrementally.
  arXiv  Detail & Related papers  (2023-04-30T22:51:31Z)
• You Only Live Once: Single-Life Reinforcement Learning [124.1738675154651]
  In many real-world situations, the goal might not be to learn a policy that can do the task repeatedly, but simply to perform a new task successfully once in a single trial. We formalize this problem setting, where an agent must complete a task within a single episode without interventions. We propose an algorithm, $Q$-weighted adversarial learning (QWALE), which employs a distribution matching strategy.
  arXiv  Detail & Related papers  (2022-10-17T09:00:11Z)
• Intrinsic fluctuations of reinforcement learning promote cooperation [0.0]
  Cooperating in social dilemma situations is vital for animals, humans, and machines. We demonstrate which and how individual elements of the multi-agent learning setting lead to cooperation.
  arXiv  Detail & Related papers  (2022-09-01T09:14:47Z)
• Deep Reinforcement Learning for Multi-Agent Interaction [14.532965827043254]
  The Autonomous Agents Research Group develops novel machine learning algorithms for autonomous systems control. This article provides a broad overview of the ongoing research portfolio of the group and discusses open problems for future directions.
  arXiv  Detail & Related papers  (2022-08-02T21:55:56Z)
• Multi-Agent Imitation Learning with Copulas [102.27052968901894]
  Multi-agent imitation learning aims to train multiple agents to perform tasks from demonstrations by learning a mapping between observations and actions. In this paper, we propose to use copula, a powerful statistical tool for capturing dependence among random variables, to explicitly model the correlation and coordination in multi-agent systems. Our proposed model is able to separately learn marginals that capture the local behavioral patterns of each individual agent, as well as a copula function that solely and fully captures the dependence structure among agents.
  arXiv  Detail & Related papers  (2021-07-10T03:49:41Z)
• Multi-Agent Interactions Modeling with Correlated Policies [53.38338964628494]
  In this paper, we cast the multi-agent interactions modeling problem into a multi-agent imitation learning framework. We develop a Decentralized Adrial Imitation Learning algorithm with Correlated policies (CoDAIL) Various experiments demonstrate that CoDAIL can better regenerate complex interactions close to the demonstrators.
  arXiv  Detail & Related papers  (2020-01-04T17:31:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.