Convergence of Actor-Critic Learning for Mean Field Games and Mean Field Control in Continuous Spaces

• URL: http://arxiv.org/abs/2511.06812v1
• Date: Mon, 10 Nov 2025 07:55:34 GMT
• Title: Convergence of Actor-Critic Learning for Mean Field Games and Mean Field Control in Continuous Spaces
• Authors: Jean-Pierre Fouque, Mathieu Laurière, Mengrui Zhang,
• Abstract summary: We establish the convergence of the deep actor-critic reinforcement learning algorithm presented in [Angiuli et al., 2023a]<n>This algorithm provides solutions to Mean Field Game (MFG) or Mean Field Control (MFC) problems depending on the ratio between two learning rates.
• Score: 2.130420850671229
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We establish the convergence of the deep actor-critic reinforcement learning algorithm presented in [Angiuli et al., 2023a] in the setting of continuous state and action spaces with an infinite discrete-time horizon. This algorithm provides solutions to Mean Field Game (MFG) or Mean Field Control (MFC) problems depending on the ratio between two learning rates: one for the value function and the other for the mean field term. In the MFC case, to rigorously identify the limit, we introduce a discretization of the state and action spaces, following the approach used in the finite-space case in [Angiuli et al., 2023b]. The convergence proofs rely on a generalization of the two-timescale framework introduced in [Borkar, 1997]. We further extend our convergence results to Mean Field Control Games, which involve locally cooperative and globally competitive populations. Finally, we present numerical experiments for linear-quadratic problems in one and two dimensions, for which explicit solutions are available.

Related papers

• Adaptive Partitioning and Learning for Stochastic Control of Diffusion Processes [3.058685580689604]
  We study reinforcement learning for controlled diffusion processes with unbounded continuous state spaces.<n>We introduce a model-based algorithm that adaptively partitions the joint state-action space.<n>This adaptive scheme balances exploration and approximation, enabling efficient learning in unbounded domains.
  arXiv  Detail & Related papers  (2025-12-17T00:52:19Z)
• Ordering-based Conditions for Global Convergence of Policy Gradient Methods [73.6366483406033]
  We prove that, for finite-arm bandits with linear function approximation, the global convergence of policy gradient (PG) methods depends on inter-related properties between the policy update and the representation.<n>Overall, these observations call into question approximation error as an appropriate quantity for characterizing the global convergence of PG methods under linear function approximation.
  arXiv  Detail & Related papers  (2025-04-02T21:06:28Z)
• Analysis of Multiscale Reinforcement Q-Learning Algorithms for Mean Field Control Games [2.3833208322103605]
  Mean Field Control Games (MFCG) represent competitive games between a large number of large collaborative groups of agents. We prove the convergence of a three-timescale Reinforcement Q-Learning (RL) algorithm to solve MFCG.
  arXiv  Detail & Related papers  (2024-05-27T10:01:52Z)
• Deep Reinforcement Learning for Infinite Horizon Mean Field Problems in Continuous Spaces [1.4999444543328293]
  We present a reinforcement learning (RL) algorithm designed to solve mean field games (MFG) and mean field control (MFC) problems in a unified manner.<n>The proposed approach pairs the actor-critic (AC) paradigm with a representation of the mean field distribution via a parameterized score function.<n>A modification of the algorithm allows us to solve mixed mean field control games (MFCGs)
  arXiv  Detail & Related papers  (2023-09-19T22:37:47Z)
• Context-aware Domain Adaptation for Time Series Anomaly Detection [69.3488037353497]
  Time series anomaly detection is a challenging task with a wide range of real-world applications. Recent efforts have been devoted to time series domain adaptation to leverage knowledge from similar domains. We propose a framework that combines context sampling and anomaly detection into a joint learning procedure.
  arXiv  Detail & Related papers  (2023-04-15T02:28:58Z)
• Tight Guarantees for Interactive Decision Making with the Decision-Estimation Coefficient [51.37720227675476]
  We introduce a new variant of the Decision-Estimation Coefficient, and use it to derive new lower bounds that improve upon prior work on three fronts. We provide upper bounds on regret that scale with the same quantity, thereby closing all but one of the gaps between upper and lower bounds in Foster et al. Our results apply to both the regret framework and PAC framework, and make use of several new analysis and algorithm design techniques that we anticipate will find broader use.
  arXiv  Detail & Related papers  (2023-01-19T18:24:08Z)
• Approximation of optimization problems with constraints through kernel Sum-Of-Squares [77.27820145069515]
  We show that pointwise inequalities are turned into equalities within a class of nonnegative kSoS functions. We also show that focusing on pointwise equality constraints enables the use of scattering inequalities to mitigate the curse of dimensionality in sampling the constraints.
  arXiv  Detail & Related papers  (2023-01-16T10:30:04Z)
• Localized Adversarial Domain Generalization [83.4195658745378]
  Adversarial domain generalization is a popular approach to domain generalization. We propose localized adversarial domain generalization with space compactness maintenance(LADG) We conduct comprehensive experiments on the Wilds DG benchmark to validate our approach.
  arXiv  Detail & Related papers  (2022-05-09T08:30:31Z)
• Lifting the Convex Conjugate in Lagrangian Relaxations: A Tractable Approach for Continuous Markov Random Fields [53.31927549039624]
  We show that a piecewise discretization preserves better contrast from existing discretization problems. We apply this theory to the problem of matching two images.
  arXiv  Detail & Related papers  (2021-07-13T12:31:06Z)
• Cross-Domain Grouping and Alignment for Domain Adaptive Semantic Segmentation [74.3349233035632]
  Existing techniques to adapt semantic segmentation networks across the source and target domains within deep convolutional neural networks (CNNs) do not consider an inter-class variation within the target domain itself or estimated category. We introduce a learnable clustering module, and a novel domain adaptation framework called cross-domain grouping and alignment. Our method consistently boosts the adaptation performance in semantic segmentation, outperforming the state-of-the-arts on various domain adaptation settings.
  arXiv  Detail & Related papers  (2020-12-15T11:36:21Z)
• Unified Reinforcement Q-Learning for Mean Field Game and Control Problems [0.0]
  We present a Reinforcement Learning (RL) algorithm to solve infinite horizon Mean Field Game (MFG) and Mean Field Control (MFC) problems. Our approach can be described as a unified two-timescale Mean Field Q-learning: The emphsame algorithm can learn either the MFG or the MFC solution by simply tuning the ratio of two learning parameters.
  arXiv  Detail & Related papers  (2020-06-24T17:45:44Z)
• On the Convergence of Overlapping Schwarz Decomposition for Nonlinear Optimal Control [7.856998585396421]
  We study the convergence properties of an overlapping decomposition algorithm for solving nonlinear Schwarz problems. We show that the algorithm exhibits local linear convergence, and that the convergence rate improves exponentially with the overlap size.
  arXiv  Detail & Related papers  (2020-05-14T00:19:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.