Evolution with Opponent-Learning Awareness

• URL: http://arxiv.org/abs/2410.17466v3
• Date: Mon, 28 Oct 2024 18:00:25 GMT
• Title: Evolution with Opponent-Learning Awareness
• Authors: Yann Bouteiller, Karthik Soma, Giovanni Beltrame,
• Abstract summary: We show how large heterogeneous populations of learning agents evolve in normal-form games. We derive a fast, parallelizable implementation of Opponent-Learning Awareness tailored for evolutionary simulations. We demonstrate our approach in simulations of 200,000 agents, evolving in the classic games of Hawk-Dove, Stag-Hunt, and Rock-Paper-Scissors.
• Score: 10.689403855269704
• License:
• Abstract: The universe involves many independent co-learning agents as an ever-evolving part of our observed environment. Yet, in practice, Multi-Agent Reinforcement Learning (MARL) applications are usually constrained to small, homogeneous populations and remain computationally intensive. In this paper, we study how large heterogeneous populations of learning agents evolve in normal-form games. We show how, under assumptions commonly made in the multi-armed bandit literature, Multi-Agent Policy Gradient closely resembles the Replicator Dynamic, and we further derive a fast, parallelizable implementation of Opponent-Learning Awareness tailored for evolutionary simulations. This enables us to simulate the evolution of very large populations made of heterogeneous co-learning agents, under both naive and advanced learning strategies. We demonstrate our approach in simulations of 200,000 agents, evolving in the classic games of Hawk-Dove, Stag-Hunt, and Rock-Paper-Scissors. Each game highlights distinct ways in which Opponent-Learning Awareness affects evolution.

Related papers

• A Survey on Self-Evolution of Large Language Models [116.54238664264928]
  Large language models (LLMs) have significantly advanced in various fields and intelligent agent applications. To address this issue, self-evolution approaches that enable LLMs to autonomously acquire, refine, and learn from experiences generated by the model itself are rapidly growing.
  arXiv  Detail & Related papers  (2024-04-22T17:43:23Z)
• Mathematics of multi-agent learning systems at the interface of game theory and artificial intelligence [0.8049333067399385]
  Evolutionary Game Theory and Artificial Intelligence are two fields that, at first glance, might seem distinct, but they have notable connections and intersections. The former focuses on the evolution of behaviors (or strategies) in a population, where individuals interact with others and update their strategies based on imitation (or social learning) The latter, meanwhile, is centered on machine learning algorithms and (deep) neural networks.
  arXiv  Detail & Related papers  (2024-03-09T17:36:54Z)
• DARLEI: Deep Accelerated Reinforcement Learning with Evolutionary Intelligence [77.78795329701367]
  We present DARLEI, a framework that combines evolutionary algorithms with parallelized reinforcement learning. We characterize DARLEI's performance under various conditions, revealing factors impacting diversity of evolved morphologies. We hope to extend DARLEI in future work to include interactions between diverse morphologies in richer environments.
  arXiv  Detail & Related papers  (2023-12-08T16:51:10Z)
• Generating Personas for Games with Multimodal Adversarial Imitation Learning [47.70823327747952]
  Reinforcement learning has been widely successful in producing agents capable of playing games at a human level. Going beyond reinforcement learning is necessary to model a wide range of human playstyles. This paper presents a novel imitation learning approach to generate multiple persona policies for playtesting.
  arXiv  Detail & Related papers  (2023-08-15T06:58:19Z)
• Supplementing Gradient-Based Reinforcement Learning with Simple Evolutionary Ideas [4.873362301533824]
  We present a simple, sample-efficient algorithm for introducing large but directed learning steps in reinforcement learning (RL) The methodology uses a population of RL agents training with a common experience buffer, with occasional crossovers and mutations of the agents in order to search efficiently through the policy space.
  arXiv  Detail & Related papers  (2023-05-10T09:46:53Z)
• Generative Adversarial Neuroevolution for Control Behaviour Imitation [3.04585143845864]
  We propose to explore whether deep neuroevolution can be used for behaviour imitation on popular simulation environments. We introduce a simple co-evolutionary adversarial generation framework, and evaluate its capabilities by evolving standard deep recurrent networks. Across all tasks, we find the final elite actor agents capable of achieving scores as high as those obtained by the pre-trained agents.
  arXiv  Detail & Related papers  (2023-04-03T16:33:22Z)
• Conditional Imitation Learning for Multi-Agent Games [89.897635970366]
  We study the problem of conditional multi-agent imitation learning, where we have access to joint trajectory demonstrations at training time. We propose a novel approach to address the difficulties of scalability and data scarcity. Our model learns a low-rank subspace over ego and partner agent strategies, then infers and adapts to a new partner strategy by interpolating in the subspace.
  arXiv  Detail & Related papers  (2022-01-05T04:40:13Z)
• From Motor Control to Team Play in Simulated Humanoid Football [56.86144022071756]
  We train teams of physically simulated humanoid avatars to play football in a realistic virtual environment. In a sequence of stages, players first learn to control a fully articulated body to perform realistic, human-like movements. They then acquire mid-level football skills such as dribbling and shooting. Finally, they develop awareness of others and play as a team, bridging the gap between low-level motor control at a timescale of milliseconds.
  arXiv  Detail & Related papers  (2021-05-25T20:17:10Z)
• Embodied Intelligence via Learning and Evolution [92.26791530545479]
  We show that environmental complexity fosters the evolution of morphological intelligence. We also show that evolution rapidly selects morphologies that learn faster. Our experiments suggest a mechanistic basis for both the Baldwin effect and the emergence of morphological intelligence.
  arXiv  Detail & Related papers  (2021-02-03T18:58:31Z)
• The Evolutionary Dynamics of Independent Learning Agents in Population Games [21.68881173635777]
  This paper presents a formal relation between processes and the dynamics of independent learning agents in population games. Using a master equation approach, we provide a novel unified framework for characterising population dynamics.
  arXiv  Detail & Related papers  (2020-06-29T14:22:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.