Towards Scaling Reinforcement Learning to Massive Populations: Learning Mean-Field Representations

• URL: http://arxiv.org/abs/2609.02928v1
• Date: Wed, 26 Aug 2026 18:54:16 GMT
• Title: Towards Scaling Reinforcement Learning to Massive Populations: Learning Mean-Field Representations
• Abstract summary: In large-population systems, the dynamics depend on an aggregate summary of the population rather than the identity of any individual.<n>We introduce a mean-field RL framework in which the rewards and transition dynamics depend on the population only through an unknown low-dimensional aggregate statistic.<n>Motivated by real-life supply-chain optimization problems, we design a one-step routing game to test the hypothesis that learning a low-dimensional population representation improves reward prediction.
• Score: 21.920988041622035
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Modern multi-agent systems are increasingly deployed at scale over large populations of agents in settings such as ad-auctions, traffic routing, and recommendation systems. The dominant approach in such settings is to optimize each agent's policy independently, treating the other agents as part of a fixed single-agent environment rather than modeling the population dynamics. In many large-population systems, the dynamics depend on an aggregate summary of the population rather than the identity of any individual. Mean-field RL exploits such structure, providing a principled framework that models each agent's environment as an explicit function of the population distribution. However, in large state-action spaces or high-dimensional control problems, modeling the population distribution is itself intractable. How can we design a scalable framework for high-dimensional control problems with large populations? This work explores this question from the perspective of representation learning. We introduce a mean-field RL framework in which the rewards and transition dynamics depend on the population only through an unknown low-dimensional aggregate statistic. We then study this framework in the offline setting and design a provable approach that learns a near-optimal policy by learning a low-dimensional representation. Motivated by real-life supply-chain optimization problems, we design a one-step routing game to test the hypothesis that learning a low-dimensional population representation improves reward prediction and Nash gap estimation relative to baselines that don't exploit this structure. We show that under a fixed neural-network parameter count and optimization budget, learning a low-dimensional population representation improves reward prediction and the equilibrium quality of the resulting policies.

Related papers

• Mean-Field Diffuser: Scaling Offline MARL to Thousands of Agents [18.152109081372544]
  We introduce MF-Diffuser, a framework that lifts trajectory planning to the Wasserstein space of trajectory distributions.<n>We show MF-Diffuser achieves the best return in the majority of settings, with the largest gains on suboptimal offline data and at extreme scales.
  arXiv  Detail & Related papers  (2026-05-28T16:32:56Z)
• AdaEvolve: Adaptive LLM Driven Zeroth-Order Optimization [61.535567824938205]
  We introduce AdaEvolve, a framework that reformulates LLM-driven evolution as a hierarchical adaptive optimization problem.<n>AdaEvolve consistently outperforms the open-ended baselines across 185 different open-ended optimization problems.
  arXiv  Detail & Related papers  (2026-02-23T18:45:31Z)
• Group size effects and collective misalignment in LLM multi-agent systems [0.0874967598360817]
  We show that collective bias is a deeper phenomenon than previously assessed.<n>We demonstrate that group size affects the dynamics in a non-linear way.<n>These findings establish group size as a key driver of multi-agent dynamics.
  arXiv  Detail & Related papers  (2025-10-25T19:45:45Z)
• Hidden Representation Clustering with Multi-Task Representation Learning towards Robust Online Budget Allocation [25.524699372749957]
  Marketing optimization, commonly formulated as an online budget allocation problem, has emerged as a pivotal factor in driving user growth.<n>This paper proposes a novel approach that solves the problem from the cluster perspective.
  arXiv  Detail & Related papers  (2025-06-01T11:09:07Z)
• MF-LLM: Simulating Population Decision Dynamics via a Mean-Field Large Language Model Framework [53.82097200295448]
  Mean-Field LLM (MF-LLM) is first to incorporate mean field theory into social simulation.<n>MF-LLM models bidirectional interactions between individuals and the population through an iterative process.<n> IB-Tune is a novel fine-tuning method inspired by the Information Bottleneck principle.
  arXiv  Detail & Related papers  (2025-04-30T12:41:51Z)
• Networked Communication for Mean-Field Games with Function Approximation and Empirical Mean-Field Estimation [59.01527054553122]
  Decentralised agents can learn equilibria in Mean-Field Games from a non-episodic run of the empirical system.<n>We introduce function approximation to the existing setting, drawing on the Munchausen Online Mirror Descent method.<n>We show theoretically that exchanging policy information helps networked agents outperform both independent and even centralised agents in function-approximation settings.
  arXiv  Detail & Related papers  (2024-08-21T13:32:46Z)
• A Single Online Agent Can Efficiently Learn Mean Field Games [16.00164239349632]
  Mean field games (MFGs) are a promising framework for modeling the behavior of large-population systems. This paper introduces a novel online single-agent model-free learning scheme, which enables a single agent to learn MFNE using online samples.
  arXiv  Detail & Related papers  (2024-05-05T16:38:04Z)
• Decentralized Learning Strategies for Estimation Error Minimization with Graph Neural Networks [86.99017195607077]
  We address the challenge of sampling and remote estimation for autoregressive Markovian processes in a wireless network with statistically-identical agents.<n>Our goal is to minimize time-average estimation error and/or age of information with decentralized scalable sampling and transmission policies.
  arXiv  Detail & Related papers  (2024-04-04T06:24:11Z)
• Efficient Model-Based Multi-Agent Mean-Field Reinforcement Learning [89.31889875864599]
  We propose an efficient model-based reinforcement learning algorithm for learning in multi-agent systems. Our main theoretical contributions are the first general regret bounds for model-based reinforcement learning for MFC. We provide a practical parametrization of the core optimization problem.
  arXiv  Detail & Related papers  (2021-07-08T18:01:02Z)
• Reinforcement Learning in Linear Quadratic Deep Structured Teams: Global Convergence of Policy Gradient Methods [0.0]
  We study the global convergence of model-based and model-free policy gradient descent and natural policy gradient descent algorithms. In such systems, agents are partitioned into a few sub-populations wherein the agents in each sub-population are coupled in the dynamics and cost function.
  arXiv  Detail & Related papers  (2020-11-29T16:02:39Z)
• BayGo: Joint Bayesian Learning and Information-Aware Graph Optimization [48.30183416069897]
  BayGo is a novel fully decentralized joint Bayesian learning and graph optimization framework. We show that our framework achieves faster convergence and higher accuracy compared to fully-connected and star topology graphs.
  arXiv  Detail & Related papers  (2020-11-09T11:16:55Z)
• Decentralized MCTS via Learned Teammate Models [89.24858306636816]
  We present a trainable online decentralized planning algorithm based on decentralized Monte Carlo Tree Search. We show that deep learning and convolutional neural networks can be employed to produce accurate policy approximators.
  arXiv  Detail & Related papers  (2020-03-19T13:10:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.