Population-Aware Imitation Learning in Mean-field Games with Common Noise

• URL: http://arxiv.org/abs/2605.03357v1
• Date: Tue, 05 May 2026 04:33:40 GMT
• Title: Population-Aware Imitation Learning in Mean-field Games with Common Noise
• Abstract summary: Mean Field Games (MFGs) provide a powerful framework for modeling the collective behavior of interacting agents.<n>We propose a numerical framework using generalized Fictitious Play and Deep Learning to compute expert population-aware policies.
• Score: 2.394379536305005
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Mean Field Games (MFGs) provide a powerful framework for modeling the collective behavior of large populations of interacting agents. In this paper, we address the problem of Imitation Learning (IL) in MFGs subject to common noise, where the population distribution evolves stochastically. This stochasticity compels agents to adopt population-aware policies to respond to aggregate shocks. We formulate two distinct learning objectives: recovering a Nash equilibrium and maximizing performance against an expert population. We investigate two imitation proxies: Behavioral Cloning (BC) and Adversarial (ADV) divergence. We then establish finite-sample error bounds showing that minimizing these proxies effectively controls both the policy's exploitability and its performance gap relative to the expert. Furthermore, we propose a numerical framework using generalized Fictitious Play and Deep Learning to compute expert population-aware policies. Through experiments on three environments we demonstrate that standard population-unaware policies fail to capture the equilibrium dynamics. Our results highlight that learning population-aware policies is crucial to avoid being misled by the randomness inherent in common noise.

Related papers

• Towards Scaling Reinforcement Learning to Massive Populations: Learning Mean-Field Representations [21.920988041622035]
  In large-population systems, the dynamics depend on an aggregate summary of the population rather than the identity of any individual.<n>We introduce a mean-field RL framework in which the rewards and transition dynamics depend on the population only through an unknown low-dimensional aggregate statistic.<n>Motivated by real-life supply-chain optimization problems, we design a one-step routing game to test the hypothesis that learning a low-dimensional population representation improves reward prediction.
  arXiv  Detail & Related papers  (2026-08-26T18:54:16Z)
• Too Correct to Learn: Reinforcement Learning on Saturated Reasoning Data [55.84428098924793]
  We propose Constrained Uniform Top-K Sampling (CUTS), a parameter-free decoding strategy enforcing structure-preserving exploration.<n>We integrate this into Mixed-CUTS, a training framework synergizing exploitative and exploratory rollouts to amplify intra-group advantage variance.<n> Notably, Mixed-CUTS improves Pass@1 accuracy on the challenging AIME25 benchmark by up to 15.1% over standard GRPO.
  arXiv  Detail & Related papers  (2026-04-20T16:43:28Z)
• LLM-Powered Social Digital Twins: A Framework for Simulating Population Behavioral Response to Policy Interventions [0.2787288702904897]
  Social Digital Twins are virtual population replicas where Large Language Models serve as cognitive engines for individual agents.<n>We instantiate this framework in the domain of pandemic response, using COVID-19 as a case study.<n>We discuss implications for policy simulation, limitations of the approach, and directions for extending LLM-based digital twins beyond pandemic response.
  arXiv  Detail & Related papers  (2026-01-03T13:25:33Z)
• Population-aware Online Mirror Descent for Mean-Field Games with Common Noise by Deep Reinforcement Learning [28.970166223191836]
  Mean Field Games (MFGs) offer a powerful framework for studying large-scale multi-agent systems.<n>Yet, learning Nash equilibria in MFGs remains a challenging problem.<n>We introduce an efficient deep reinforcement learning (DRL) algorithm to achieve population-dependent Nash equilibria.
  arXiv  Detail & Related papers  (2025-09-03T05:33:46Z)
• COMBO-Grasp: Learning Constraint-Based Manipulation for Bimanual Occluded Grasping [56.907940167333656]
  Occluded robot grasping is where the desired grasp poses are kinematically infeasible due to environmental constraints such as surface collisions.<n>Traditional robot manipulation approaches struggle with the complexity of non-prehensile or bimanual strategies commonly used by humans.<n>We introduce Constraint-based Manipulation for Bimanual Occluded Grasping (COMBO-Grasp), a learning-based approach which leverages two coordinated policies.
  arXiv  Detail & Related papers  (2025-02-12T01:31:01Z)
• Learnable Behavior Control: Breaking Atari Human World Records via Sample-Efficient Behavior Selection [80.35510218548693]
  We propose a general framework called Learnable Behavioral Control (LBC) to address the limitation.<n>Our agents have achieved 10077.52% mean human normalized score and surpassed 24 human world records within 1B training frames.
  arXiv  Detail & Related papers  (2023-05-09T08:00:23Z)
• Safe Multi-agent Learning via Trapping Regions [89.24858306636816]
  We apply the concept of trapping regions, known from qualitative theory of dynamical systems, to create safety sets in the joint strategy space for decentralized learning. We propose a binary partitioning algorithm for verification that candidate sets form trapping regions in systems with known learning dynamics, and a sampling algorithm for scenarios where learning dynamics are not known.
  arXiv  Detail & Related papers  (2023-02-27T14:47:52Z)
• Population-size-Aware Policy Optimization for Mean-Field Games [34.80183622480149]
  We study how the optimal policies of agents evolve with the number of agents (population size) in mean-field games. We propose Population-size-Aware Policy Optimization (PAPO), which unifies two natural options (augmentation and hypernetwork) and significantly better performance. PAPO consists of three components: i) the population-size encoding which transforms the original value of population size to an equivalent encoding to avoid training collapse, ii) a hypernetwork to generate a distinct policy for each game conditioned on the population size, andiii) the population size as an additional input to the generated policy.
  arXiv  Detail & Related papers  (2023-02-07T10:16:00Z)
• Robust Reinforcement Learning using Adversarial Populations [118.73193330231163]
  Reinforcement Learning (RL) is an effective tool for controller design but can struggle with issues of robustness. We show that using a single adversary does not consistently yield robustness to dynamics variations under standard parametrizations of the adversary. We propose a population-based augmentation to the Robust RL formulation in which we randomly initialize a population of adversaries and sample from the population uniformly during training.
  arXiv  Detail & Related papers  (2020-08-04T20:57:32Z)
• Off-policy Evaluation in Infinite-Horizon Reinforcement Learning with Latent Confounders [62.54431888432302]
  We study an OPE problem in an infinite-horizon, ergodic Markov decision process with unobserved confounders. We show how, given only a latent variable model for states and actions, policy value can be identified from off-policy data.
  arXiv  Detail & Related papers  (2020-07-27T22:19:01Z)
• Policy Evaluation and Seeking for Multi-Agent Reinforcement Learning via Best Response [15.149039407681945]
  We adopt strict best response dynamics to model selfish behaviors at a meta-level for multi-agent reinforcement learning. Our approach is more compatible with single-agent reinforcement learning than alpha-rank which relies on weakly better responses.
  arXiv  Detail & Related papers  (2020-06-17T01:17:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.