Emergent Prosociality in Multi-Agent Games Through Gifting

• URL: http://arxiv.org/abs/2105.06593v1
• Date: Thu, 13 May 2021 23:28:30 GMT
• Title: Emergent Prosociality in Multi-Agent Games Through Gifting
• Authors: Woodrow Z. Wang, Mark Beliaev, Erdem B{\i}y{\i}k, Daniel A. Lazar, Ramtin Pedarsani, Dorsa Sadigh
• Abstract summary: Reinforcement learning algorithms often suffer from converging to socially less desirable equilibria when multiple equilibria exist. We propose using a less restrictive peer-rewarding mechanism, gifting, that guides the agents toward more socially desirable equilibria. We employ a theoretical framework that captures the benefit of gifting in converging to the prosocial equilibrium.
• Score: 14.943238230772264
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Coordination is often critical to forming prosocial behaviors -- behaviors that increase the overall sum of rewards received by all agents in a multi-agent game. However, state of the art reinforcement learning algorithms often suffer from converging to socially less desirable equilibria when multiple equilibria exist. Previous works address this challenge with explicit reward shaping, which requires the strong assumption that agents can be forced to be prosocial. We propose using a less restrictive peer-rewarding mechanism, gifting, that guides the agents toward more socially desirable equilibria while allowing agents to remain selfish and decentralized. Gifting allows each agent to give some of their reward to other agents. We employ a theoretical framework that captures the benefit of gifting in converging to the prosocial equilibrium by characterizing the equilibria's basins of attraction in a dynamical system. With gifting, we demonstrate increased convergence of high risk, general-sum coordination games to the prosocial equilibrium both via numerical analysis and experiments.

Related papers

• Learning to Balance Altruism and Self-interest Based on Empathy in Mixed-Motive Games [47.8980880888222]
  Multi-agent scenarios often involve mixed motives, demanding altruistic agents capable of self-protection against potential exploitation. We propose LASE Learning to balance Altruism and Self-interest based on Empathy. LASE allocates a portion of its rewards to co-players as gifts, with this allocation adapting dynamically based on the social relationship.
  arXiv  Detail & Related papers  (2024-10-10T12:30:56Z)
• Linear Convergence of Independent Natural Policy Gradient in Games with Entropy Regularization [12.612009339150504]
  This work focuses on the entropy-regularized independent natural policy gradient (NPG) algorithm in multi-agent reinforcement learning. We show that, under sufficient entropy regularization, the dynamics of this system converge at a linear rate to the quantal response equilibrium (QRE)
  arXiv  Detail & Related papers  (2024-05-04T22:48:53Z)
• A Black-box Approach for Non-stationary Multi-agent Reinforcement Learning [53.83345471268163]
  We investigate learning the equilibria in non-stationary multi-agent systems. We show how to test for various types of equilibria by a black-box reduction to single-agent learning.
  arXiv  Detail & Related papers  (2023-06-12T23:48:24Z)
• Game-Theoretical Perspectives on Active Equilibria: A Preferred Solution Concept over Nash Equilibria [61.093297204685264]
  An effective approach in multiagent reinforcement learning is to consider the learning process of agents and influence their future policies. This new solution concept is general such that standard solution concepts, such as a Nash equilibrium, are special cases of active equilibria. We analyze active equilibria from a game-theoretic perspective by closely studying examples where Nash equilibria are known.
  arXiv  Detail & Related papers  (2022-10-28T14:45:39Z)
• How Bad is Selfish Driving? Bounding the Inefficiency of Equilibria in Urban Driving Games [64.71476526716668]
  We study the (in)efficiency of any equilibrium players might agree to play. We obtain guarantees that refine existing bounds on the Price of Anarchy. Although the obtained guarantees concern open-loop trajectories, we observe efficient equilibria even when agents employ closed-loop policies.
  arXiv  Detail & Related papers  (2022-10-24T09:32:40Z)
• ELIGN: Expectation Alignment as a Multi-Agent Intrinsic Reward [29.737986509769808]
  We propose a self-supervised intrinsic reward ELIGN - expectation alignment. Similar to how animals collaborate in a decentralized manner with those in their vicinity, agents trained with expectation alignment learn behaviors that match their neighbors' expectations. We show that agent coordination improves through expectation alignment because agents learn to divide tasks amongst themselves, break coordination symmetries, and confuse adversaries.
  arXiv  Detail & Related papers  (2022-10-09T22:24:44Z)
• Formal Contracts Mitigate Social Dilemmas in Multi-Agent RL [4.969697978555126]
  Multi-agent Reinforcement Learning (MARL) is a powerful tool for training autonomous agents acting independently in a common environment. MARL can lead to sub-optimal behavior when individual incentives and group incentives diverge. We propose an augmentation to a Markov game where agents voluntarily agree to binding transfers of reward, under pre-specified conditions.
  arXiv  Detail & Related papers  (2022-08-22T17:42:03Z)
• Inducing Equilibria via Incentives: Simultaneous Design-and-Play Finds Global Optima [114.31577038081026]
  We propose an efficient method that tackles the designer's and agents' problems simultaneously in a single loop. Although the designer does not solve the equilibrium problem repeatedly, it can anticipate the overall influence of the incentives on the agents. We prove that the algorithm converges to the global optima at a sublinear rate for a broad class of games.
  arXiv  Detail & Related papers  (2021-10-04T06:53:59Z)
• Learning to Incentivize Other Learning Agents [73.03133692589532]
  We show how to equip RL agents with the ability to give rewards directly to other agents, using a learned incentive function. Such agents significantly outperform standard RL and opponent-shaping agents in challenging general-sum Markov games. Our work points toward more opportunities and challenges along the path to ensure the common good in a multi-agent future.
  arXiv  Detail & Related papers  (2020-06-10T20:12:38Z)
• Inducing Cooperative behaviour in Sequential-Social dilemmas through Multi-Agent Reinforcement Learning using Status-Quo Loss [16.016452248865132]
  In social dilemma situations, individual rationality leads to sub-optimal group outcomes. Deep Reinforcement Learning agents trained to optimize individual rewards converge to selfish, mutually harmful behavior. We show how agents trained with SQLoss evolve cooperative behavior in several social dilemma matrix games.
  arXiv  Detail & Related papers  (2020-01-15T18:10:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.