Generalization in Mean Field Games by Learning Master Policies
- URL: http://arxiv.org/abs/2109.09717v1
- Date: Mon, 20 Sep 2021 17:47:34 GMT
- Title: Generalization in Mean Field Games by Learning Master Policies
- Authors: Sarah Perrin and Mathieu Lauri\`ere and Julien P\'erolat and Romuald
\'Elie and Matthieu Geist and Olivier Pietquin
- Abstract summary: Mean Field Games (MFGs) can potentially scale multi-agent systems to extremely large populations of agents.
We study how to leverage generalization properties to learn policies enabling a typical agent to behave optimally against any population distribution.
- Score: 34.67098179276852
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Mean Field Games (MFGs) can potentially scale multi-agent systems to
extremely large populations of agents. Yet, most of the literature assumes a
single initial distribution for the agents, which limits the practical
applications of MFGs. Machine Learning has the potential to solve a wider
diversity of MFG problems thanks to generalizations capacities. We study how to
leverage these generalization properties to learn policies enabling a typical
agent to behave optimally against any population distribution. In reference to
the Master equation in MFGs, we coin the term ``Master policies'' to describe
them and we prove that a single Master policy provides a Nash equilibrium,
whatever the initial distribution. We propose a method to learn such Master
policies. Our approach relies on three ingredients: adding the current
population distribution as part of the observation, approximating Master
policies with neural networks, and training via Reinforcement Learning and
Fictitious Play. We illustrate on numerical examples not only the efficiency of
the learned Master policy but also its generalization capabilities beyond the
distributions used for training.
Related papers
- A Single Online Agent Can Efficiently Learn Mean Field Games [16.00164239349632]
Mean field games (MFGs) are a promising framework for modeling the behavior of large-population systems.
This paper introduces a novel online single-agent model-free learning scheme, which enables a single agent to learn MFNE using online samples.
arXiv Detail & Related papers (2024-05-05T16:38:04Z) - Acquiring Diverse Skills using Curriculum Reinforcement Learning with Mixture of Experts [58.220879689376744]
Reinforcement learning (RL) is a powerful approach for acquiring a good-performing policy.
We propose textbfDiverse textbfSkill textbfLearning (Di-SkilL) for learning diverse skills.
We show on challenging robot simulation tasks that Di-SkilL can learn diverse and performant skills.
arXiv Detail & Related papers (2024-03-11T17:49:18Z) - Population-aware Online Mirror Descent for Mean-Field Games by Deep
Reinforcement Learning [43.004209289015975]
Mean Field Games (MFGs) have the ability to handle large-scale multi-agent systems.
We propose a deep reinforcement learning (DRL) algorithm that achieves population-dependent Nash equilibrium.
arXiv Detail & Related papers (2024-03-06T08:55:34Z) - Is Inverse Reinforcement Learning Harder than Standard Reinforcement
Learning? A Theoretical Perspective [55.36819597141271]
Inverse Reinforcement Learning (IRL) -- the problem of learning reward functions from demonstrations of an emphexpert policy -- plays a critical role in developing intelligent systems.
This paper provides the first line of efficient IRL in vanilla offline and online settings using samples and runtime.
As an application, we show that the learned rewards can emphtransfer to another target MDP with suitable guarantees.
arXiv Detail & Related papers (2023-11-29T00:09:01Z) - Regularization of the policy updates for stabilizing Mean Field Games [0.2348805691644085]
This work studies non-cooperative Multi-Agent Reinforcement Learning (MARL)
MARL where multiple agents interact in the same environment and whose goal is to maximize the individual returns.
We name our algorithm Mean Field Proximal Policy Optimization (MF-PPO), and we empirically show the effectiveness of our method in the OpenSpiel framework.
arXiv Detail & Related papers (2023-04-04T05:45:42Z) - Learning Meta Representations for Agents in Multi-Agent Reinforcement
Learning [12.170248966278281]
In multi-agent reinforcement learning, behaviors that agents learn in a single Markov Game (MG) are typically confined to the given agent number.
In this work, our focus is on creating agents that can generalize across population-varying MGs.
Instead of learning a unimodal policy, each agent learns a policy set comprising effective strategies across a variety of games.
arXiv Detail & Related papers (2021-08-30T04:30:53Z) - DisCo RL: Distribution-Conditioned Reinforcement Learning for
General-Purpose Policies [116.12670064963625]
We develop an off-policy algorithm called distribution-conditioned reinforcement learning (DisCo RL) to efficiently learn contextual policies.
We evaluate DisCo RL on a variety of robot manipulation tasks and find that it significantly outperforms prior methods on tasks that require generalization to new goal distributions.
arXiv Detail & Related papers (2021-04-23T16:51:58Z) - Imitation Learning from MPC for Quadrupedal Multi-Gait Control [63.617157490920505]
We present a learning algorithm for training a single policy that imitates multiple gaits of a walking robot.
We use and extend MPC-Net, which is an Imitation Learning approach guided by Model Predictive Control.
We validate our approach on hardware and show that a single learned policy can replace its teacher to control multiple gaits.
arXiv Detail & Related papers (2021-03-26T08:48:53Z) - FACMAC: Factored Multi-Agent Centralised Policy Gradients [103.30380537282517]
We propose FACtored Multi-Agent Centralised policy gradients (FACMAC)
It is a new method for cooperative multi-agent reinforcement learning in both discrete and continuous action spaces.
We evaluate FACMAC on variants of the multi-agent particle environments, a novel multi-agent MuJoCo benchmark, and a challenging set of StarCraft II micromanagement tasks.
arXiv Detail & Related papers (2020-03-14T21:29:09Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.