On Imitation in Mean-field Games

• URL: http://arxiv.org/abs/2306.14799v1
• Date: Mon, 26 Jun 2023 15:58:13 GMT
• Title: On Imitation in Mean-field Games
• Authors: Giorgia Ramponi, Pavel Kolev, Olivier Pietquin, Niao He, Mathieu Lauri\`ere, Matthieu Geist
• Abstract summary: We explore the problem of imitation learning (IL) in the context of mean-field games (MFGs) We show that when only the reward depends on the population distribution, IL in MFGs can be reduced to single-agent IL with similar guarantees. We propose a new adversarial formulation where the reinforcement learning problem is replaced by a mean-field control problem.
• Score: 53.27734434016737
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We explore the problem of imitation learning (IL) in the context of mean-field games (MFGs), where the goal is to imitate the behavior of a population of agents following a Nash equilibrium policy according to some unknown payoff function. IL in MFGs presents new challenges compared to single-agent IL, particularly when both the reward function and the transition kernel depend on the population distribution. In this paper, departing from the existing literature on IL for MFGs, we introduce a new solution concept called the Nash imitation gap. Then we show that when only the reward depends on the population distribution, IL in MFGs can be reduced to single-agent IL with similar guarantees. However, when the dynamics is population-dependent, we provide a novel upper-bound that suggests IL is harder in this setting. To address this issue, we propose a new adversarial formulation where the reinforcement learning problem is replaced by a mean-field control (MFC) problem, suggesting progress in IL within MFGs may have to build upon MFC.

Related papers

• A Single Online Agent Can Efficiently Learn Mean Field Games [16.00164239349632]
  Mean field games (MFGs) are a promising framework for modeling the behavior of large-population systems. This paper introduces a novel online single-agent model-free learning scheme, which enables a single agent to learn MFNE using online samples.
  arXiv  Detail & Related papers  (2024-05-05T16:38:04Z)
• Robust Multi-Agent Reinforcement Learning via Adversarial Regularization: Theoretical Foundation and Stable Algorithms [79.61176746380718]
  Multi-Agent Reinforcement Learning (MARL) has shown promising results across several domains. MARL policies often lack robustness and are sensitive to small changes in their environment. We show that we can gain robustness by controlling a policy's Lipschitz constant. We propose a new robust MARL framework, ERNIE, that promotes the Lipschitz continuity of the policies.
  arXiv  Detail & Related papers  (2023-10-16T20:14:06Z)
• Decentralized Online Learning in Task Assignment Games for Mobile Crowdsensing [55.07662765269297]
  A mobile crowdsensing platform (MCSP) sequentially publishes sensing tasks to the available mobile units (MUs) that signal their willingness to participate in a task by sending sensing offers back to the MCSP. A stable task assignment must address two challenges: the MCSP's and MUs' conflicting goals, and the uncertainty about the MUs' required efforts and preferences. To overcome these challenges a novel decentralized approach combining matching theory and online learning, called collision-avoidance multi-armed bandit with strategic free sensing (CA-MAB-SFS) is proposed.
  arXiv  Detail & Related papers  (2023-09-19T13:07:15Z)
• Regularization of the policy updates for stabilizing Mean Field Games [0.2348805691644085]
  This work studies non-cooperative Multi-Agent Reinforcement Learning (MARL) MARL where multiple agents interact in the same environment and whose goal is to maximize the individual returns. We name our algorithm Mean Field Proximal Policy Optimization (MF-PPO), and we empirically show the effectiveness of our method in the OpenSpiel framework.
  arXiv  Detail & Related papers  (2023-04-04T05:45:42Z)
• Individual-Level Inverse Reinforcement Learning for Mean Field Games [16.79251229846642]
  Mean Field IRL (MFIRL) is the first dedicated IRL framework for MFGs that can handle both cooperative and non-cooperative environments. We develop a practical algorithm effective for MFGs with unknown dynamics.
  arXiv  Detail & Related papers  (2022-02-13T20:35:01Z)
• Reinforcement Learning for Mean Field Games, with Applications to Economics [0.0]
  Mean field games (MFG) and mean field control problems (MFC) are frameworks to study Nash equilibria or social optima in games with a continuum of agents. We present a two timescale approach with RL for MFG and MFC, which relies on a unified Q-learning algorithm.
  arXiv  Detail & Related papers  (2021-06-25T16:45:04Z)
• Concave Utility Reinforcement Learning: the Mean-field Game viewpoint [42.403650997341806]
  Concave Utility Reinforcement Learning (CURL) extends RL from linear to concave utilities in the occupancy measure induced by the agent's policy. This more general paradigm invalidates the classical Bellman equations, and calls for new algorithms. We show that CURL is a subclass of Mean-field Games (MFGs)
  arXiv  Detail & Related papers  (2021-06-07T16:51:07Z)
• Dealing with Non-Stationarity in Multi-Agent Reinforcement Learning via Trust Region Decomposition [52.06086375833474]
  Non-stationarity is one thorny issue in multi-agent reinforcement learning. We introduce a $delta$-stationarity measurement to explicitly model the stationarity of a policy sequence. We propose a trust region decomposition network based on message passing to estimate the joint policy divergence.
  arXiv  Detail & Related papers  (2021-02-21T14:46:50Z)
• Global Convergence of Policy Gradient for Linear-Quadratic Mean-Field Control/Game in Continuous Time [109.06623773924737]
  We study the policy gradient method for the linear-quadratic mean-field control and game. We show that it converges to the optimal solution at a linear rate, which is verified by a synthetic simulation.
  arXiv  Detail & Related papers  (2020-08-16T06:34:11Z)
• Breaking the Curse of Many Agents: Provable Mean Embedding Q-Iteration for Mean-Field Reinforcement Learning [135.64775986546505]
  We exploit the symmetry of agents in multi-agent reinforcement learning (MARL) We propose MF-FQI algorithm that solves the mean-field MARL and establishes a non-asymptotic analysis for MF-FQI algorithm. We highlight that MF-FQI algorithm enjoys a "blessing of many agents" property in the sense that a larger number of observed agents improves the performance of MF-FQI algorithm.
  arXiv  Detail & Related papers  (2020-06-21T21:45:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.