Stability of Multi-Agent Learning in Competitive Networks: Delaying the Onset of Chaos

• URL: http://arxiv.org/abs/2312.11943v1
• Date: Tue, 19 Dec 2023 08:41:06 GMT
• Title: Stability of Multi-Agent Learning in Competitive Networks: Delaying the Onset of Chaos
• Authors: Aamal Hussain and Francesco Belardinelli
• Abstract summary: Behaviour of multi-agent learning in competitive network games is often studied within the context of zero-sum games. We study the Q-Learning dynamics, a popular model of exploration and exploitation in multi-agent learning. We find that the stability of Q-Learning is explicitly dependent only on the network connectivity rather than the total number of agents.
• Score: 9.220952628571812
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The behaviour of multi-agent learning in competitive network games is often studied within the context of zero-sum games, in which convergence guarantees may be obtained. However, outside of this class the behaviour of learning is known to display complex behaviours and convergence cannot be always guaranteed. Nonetheless, in order to develop a complete picture of the behaviour of multi-agent learning in competitive settings, the zero-sum assumption must be lifted. Motivated by this we study the Q-Learning dynamics, a popular model of exploration and exploitation in multi-agent learning, in competitive network games. We determine how the degree of competition, exploration rate and network connectivity impact the convergence of Q-Learning. To study generic competitive games, we parameterise network games in terms of correlations between agent payoffs and study the average behaviour of the Q-Learning dynamics across all games drawn from a choice of this parameter. This statistical approach establishes choices of parameters for which Q-Learning dynamics converge to a stable fixed point. Differently to previous works, we find that the stability of Q-Learning is explicitly dependent only on the network connectivity rather than the total number of agents. Our experiments validate these findings and show that, under certain network structures, the total number of agents can be increased without increasing the likelihood of unstable or chaotic behaviours.

Related papers

• DCIR: Dynamic Consistency Intrinsic Reward for Multi-Agent Reinforcement Learning [84.22561239481901]
  We propose a new approach that enables agents to learn whether their behaviors should be consistent with that of other agents. We evaluate DCIR in multiple environments including Multi-agent Particle, Google Research Football and StarCraft II Micromanagement.
  arXiv  Detail & Related papers  (2023-12-10T06:03:57Z)
• Inverse Factorized Q-Learning for Cooperative Multi-agent Imitation Learning [13.060023718506917]
  imitation learning (IL) is a problem of learning to mimic expert behaviors from demonstrations in cooperative multi-agent systems. We introduce a novel multi-agent IL algorithm designed to address these challenges. Our approach enables the centralized learning by leveraging mixing networks to aggregate decentralized Q functions.
  arXiv  Detail & Related papers  (2023-10-10T17:11:20Z)
• Stability of Multi-Agent Learning: Convergence in Network Games with Many Players [34.33866138792406]
  We study Q-Learning dynamics and determine a sufficient condition for the dynamics to converge to a unique equilibrium in any network game. We evaluate this result on a number of representative network games and show that, under suitable network conditions, stable learning dynamics can be achieved with an arbitrary number of agents.
  arXiv  Detail & Related papers  (2023-07-26T02:45:02Z)
• Safe Multi-agent Learning via Trapping Regions [89.24858306636816]
  We apply the concept of trapping regions, known from qualitative theory of dynamical systems, to create safety sets in the joint strategy space for decentralized learning. We propose a binary partitioning algorithm for verification that candidate sets form trapping regions in systems with known learning dynamics, and a sampling algorithm for scenarios where learning dynamics are not known.
  arXiv  Detail & Related papers  (2023-02-27T14:47:52Z)
• A Unifying Perspective on Multi-Calibration: Game Dynamics for Multi-Objective Learning [63.20009081099896]
  We provide a unifying framework for the design and analysis of multicalibrated predictors. We exploit connections to game dynamics to achieve state-of-the-art guarantees for a diverse set of multicalibration learning problems.
  arXiv  Detail & Related papers  (2023-02-21T18:24:17Z)
• Asymptotic Convergence and Performance of Multi-Agent Q-Learning Dynamics [38.5932141555258]
  We study the dynamics of smooth Q-Learning, a popular reinforcement learning algorithm. We show a sufficient condition on the rate of exploration such that the Q-Learning dynamics is guaranteed to converge to a unique equilibrium in any game.
  arXiv  Detail & Related papers  (2023-01-23T18:39:11Z)
• Competing Mutual Information Constraints with Stochastic Competition-based Activations for Learning Diversified Representations [5.981521556433909]
  This work aims to address the long-established problem of learning diversified representations. We combine information-theoretic arguments with competition-based activations. As we experimentally show, the resulting networks yield significant discnative representation learning abilities.
  arXiv  Detail & Related papers  (2022-01-10T20:12:13Z)
• Exploration-Exploitation in Multi-Agent Competition: Convergence with Bounded Rationality [21.94743452608215]
  We study smooth Q-learning, a prototypical learning model that captures the balance between game rewards and exploration costs. We show that Q-learning always converges to the unique quantal-response equilibrium (QRE), the standard solution concept for games under bounded rationality.
  arXiv  Detail & Related papers  (2021-06-24T11:43:38Z)
• On Information Asymmetry in Competitive Multi-Agent Reinforcement Learning: Convergence and Optimality [78.76529463321374]
  We study the system of interacting non-cooperative two Q-learning agents. We show that this information asymmetry can lead to a stable outcome of population learning.
  arXiv  Detail & Related papers  (2020-10-21T11:19:53Z)
• Randomized Entity-wise Factorization for Multi-Agent Reinforcement Learning [59.62721526353915]
  Multi-agent settings in the real world often involve tasks with varying types and quantities of agents and non-agent entities. Our method aims to leverage these commonalities by asking the question: What is the expected utility of each agent when only considering a randomly selected sub-group of its observed entities?''
  arXiv  Detail & Related papers  (2020-06-07T18:28:41Z)
• Learning from Learners: Adapting Reinforcement Learning Agents to be Competitive in a Card Game [71.24825724518847]
  We present a study on how popular reinforcement learning algorithms can be adapted to learn and to play a real-world implementation of a competitive multiplayer card game. We propose specific training and validation routines for the learning agents, in order to evaluate how the agents learn to be competitive and explain how they adapt to each others' playing style.
  arXiv  Detail & Related papers  (2020-04-08T14:11:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.