Population-aware Online Mirror Descent for Mean-Field Games with Common Noise by Deep Reinforcement Learning

• URL: http://arxiv.org/abs/2509.03030v1
• Date: Wed, 03 Sep 2025 05:33:46 GMT
• Title: Population-aware Online Mirror Descent for Mean-Field Games with Common Noise by Deep Reinforcement Learning
• Abstract summary: Mean Field Games (MFGs) offer a powerful framework for studying large-scale multi-agent systems.<n>Yet, learning Nash equilibria in MFGs remains a challenging problem.<n>We introduce an efficient deep reinforcement learning (DRL) algorithm to achieve population-dependent Nash equilibria.
• Score: 28.970166223191836
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Mean Field Games (MFGs) offer a powerful framework for studying large-scale multi-agent systems. Yet, learning Nash equilibria in MFGs remains a challenging problem, particularly when the initial distribution is unknown or when the population is subject to common noise. In this paper, we introduce an efficient deep reinforcement learning (DRL) algorithm designed to achieve population-dependent Nash equilibria without relying on averaging or historical sampling, inspired by Munchausen RL and Online Mirror Descent. The resulting policy is adaptable to various initial distributions and sources of common noise. Through numerical experiments on seven canonical examples, we demonstrate that our algorithm exhibits superior convergence properties compared to state-of-the-art algorithms, particularly a DRL version of Fictitious Play for population-dependent policies. The performance in the presence of common noise underscores the robustness and adaptability of our approach.

Related papers

• Towards Scaling Reinforcement Learning to Massive Populations: Learning Mean-Field Representations [21.920988041622035]
  In large-population systems, the dynamics depend on an aggregate summary of the population rather than the identity of any individual.<n>We introduce a mean-field RL framework in which the rewards and transition dynamics depend on the population only through an unknown low-dimensional aggregate statistic.<n>Motivated by real-life supply-chain optimization problems, we design a one-step routing game to test the hypothesis that learning a low-dimensional population representation improves reward prediction.
  arXiv  Detail & Related papers  (2026-08-26T18:54:16Z)
• Population-Aware Imitation Learning in Mean-field Games with Common Noise [2.394379536305005]
  Mean Field Games (MFGs) provide a powerful framework for modeling the collective behavior of interacting agents.<n>We propose a numerical framework using generalized Fictitious Play and Deep Learning to compute expert population-aware policies.
  arXiv  Detail & Related papers  (2026-05-05T04:33:40Z)
• Too Correct to Learn: Reinforcement Learning on Saturated Reasoning Data [55.84428098924793]
  We propose Constrained Uniform Top-K Sampling (CUTS), a parameter-free decoding strategy enforcing structure-preserving exploration.<n>We integrate this into Mixed-CUTS, a training framework synergizing exploitative and exploratory rollouts to amplify intra-group advantage variance.<n> Notably, Mixed-CUTS improves Pass@1 accuracy on the challenging AIME25 benchmark by up to 15.1% over standard GRPO.
  arXiv  Detail & Related papers  (2026-04-20T16:43:28Z)
• Solving Continuous Mean Field Games: Deep Reinforcement Learning for Non-Stationary Dynamics [4.4685491639611685]
  Mean field games (MFGs) have emerged as a powerful framework for modeling interactions in large-scale multi-agent systems.<n>This paper introduces a novel deep reinforcement learning (DRL) algorithm specifically designed for non-stationary continuous MFGs.
  arXiv  Detail & Related papers  (2025-10-25T04:50:52Z)
• DISCO Balances the Scales: Adaptive Domain- and Difficulty-Aware Reinforcement Learning on Imbalanced Data [65.09939942413651]
  We propose a principled extension to GRPO that addresses inter-group imbalance with two key innovations.<n> Domain-aware reward scaling counteracts frequency bias by reweighting optimization based on domain prevalence.<n>Difficulty-aware reward scaling leverages prompt-level self-consistency to identify and prioritize uncertain prompts that offer greater learning value.
  arXiv  Detail & Related papers  (2025-05-21T03:43:29Z)
• Provably Efficient Information-Directed Sampling Algorithms for Multi-Agent Reinforcement Learning [50.92957910121088]
  This work designs and analyzes a novel set of algorithms for multi-agent reinforcement learning (MARL) based on the principle of information-directed sampling (IDS) For episodic two-player zero-sum MGs, we present three sample-efficient algorithms for learning Nash equilibrium. We extend Reg-MAIDS to multi-player general-sum MGs and prove that it can learn either the Nash equilibrium or coarse correlated equilibrium in a sample efficient manner.
  arXiv  Detail & Related papers  (2024-04-30T06:48:56Z)
• Population-aware Online Mirror Descent for Mean-Field Games by Deep Reinforcement Learning [43.004209289015975]
  Mean Field Games (MFGs) have the ability to handle large-scale multi-agent systems. We propose a deep reinforcement learning (DRL) algorithm that achieves population-dependent Nash equilibrium.
  arXiv  Detail & Related papers  (2024-03-06T08:55:34Z)
• Model-Based RL for Mean-Field Games is not Statistically Harder than Single-Agent RL [57.745700271150454]
  We study the sample complexity of reinforcement learning in Mean-Field Games (MFGs) with model-based function approximation. We introduce the Partial Model-Based Eluder Dimension (P-MBED), a more effective notion to characterize the model class complexity.
  arXiv  Detail & Related papers  (2024-02-08T14:54:47Z)
• Heavy-tailed denoising score matching [5.371337604556311]
  We develop an iterative noise scaling algorithm to consistently initialise the multiple levels of noise in Langevin dynamics. On the practical side, our use of heavy-tailed DSM leads to improved score estimation, controllable sampling convergence, and more balanced unconditional generative performance for imbalanced datasets.
  arXiv  Detail & Related papers  (2021-12-17T22:04:55Z)
• Signatured Deep Fictitious Play for Mean Field Games with Common Noise [0.0]
  Existing deep learning methods for solving mean-field games (MFGs) with common noise fix the sampling common noise paths and then solve the corresponding MFGs. We propose a novel single-loop algorithm, named signatured deep fictitious play, by which we can work with the unfixed common noise setup to avoid the nested-loop structure. The proposed algorithm can accurately capture the effect of common uncertainty changes on mean-field equilibria without further training of neural networks.
  arXiv  Detail & Related papers  (2021-06-06T23:09:46Z)
• Mean Field Games Flock! The Reinforcement Learning Way [34.67098179276852]
  We present a method enabling a large number of agents to learn how to flock. This is a natural behavior observed in large populations of animals. We show numerically that our algorithm learn multi-group or high-dimensional flocking with obstacles.
  arXiv  Detail & Related papers  (2021-05-17T15:17:36Z)
• Scaling up Mean Field Games with Online Mirror Descent [55.36153467919289]
  We address scaling up equilibrium computation in Mean Field Games (MFGs) using Online Mirror Descent (OMD) We show that continuous-time OMD provably converges to a Nash equilibrium under a natural and well-motivated set of monotonicity assumptions. A thorough experimental investigation on various single and multi-population MFGs shows that OMD outperforms traditional algorithms such as Fictitious Play (FP)
  arXiv  Detail & Related papers  (2021-02-28T21:28:36Z)
• Robust Reinforcement Learning using Adversarial Populations [118.73193330231163]
  Reinforcement Learning (RL) is an effective tool for controller design but can struggle with issues of robustness. We show that using a single adversary does not consistently yield robustness to dynamics variations under standard parametrizations of the adversary. We propose a population-based augmentation to the Robust RL formulation in which we randomly initialize a population of adversaries and sample from the population uniformly during training.
  arXiv  Detail & Related papers  (2020-08-04T20:57:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.