Kernel Based Maximum Entropy Inverse Reinforcement Learning for Mean-Field Games

• URL: http://arxiv.org/abs/2507.14529v1
• Date: Sat, 19 Jul 2025 08:06:52 GMT
• Title: Kernel Based Maximum Entropy Inverse Reinforcement Learning for Mean-Field Games
• Abstract summary: We consider the maximum causal entropy inverse reinforcement learning problem for stationary mean-field games.<n>This allows the inference of rich and potentially nonlinear reward structures directly from expert demonstrations.<n>We demonstrate the effectiveness of our method on a mean-field traffic routing game, where it accurately recovers expert behavior.
• Score: 2.867517731896504
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We consider the maximum causal entropy inverse reinforcement learning problem for infinite-horizon stationary mean-field games, in which we model the unknown reward function within a reproducing kernel Hilbert space. This allows the inference of rich and potentially nonlinear reward structures directly from expert demonstrations, in contrast to most existing inverse reinforcement learning approaches for mean-field games that typically restrict the reward function to a linear combination of a fixed finite set of basis functions. We also focus on the infinite-horizon cost structure, whereas prior studies primarily rely on finite-horizon formulations. We introduce a Lagrangian relaxation to this maximum causal entropy inverse reinforcement learning problem that enables us to reformulate it as an unconstrained log-likelihood maximization problem, and obtain a solution \lk{via} a gradient ascent algorithm. To illustrate the theoretical consistency of the algorithm, we establish the smoothness of the log-likelihood objective by proving the Fr\'echet differentiability of the related soft Bellman operators with respect to the parameters in the reproducing kernel Hilbert space. We demonstrate the effectiveness of our method on a mean-field traffic routing game, where it accurately recovers expert behavior.

Related papers

• Maximum Entropy Inverse Reinforcement Learning for Mean-Field Games with Average Reward [0.0]
  We study inverse reinforcement learning for mean-field games (MFGs) under an average-reward criterion.<n>We formulate the inverse problem by enforcing consistency with the expert mean-field term and long-run feature expectations.<n>Two numerical examples, a malware-spread MFG and an RKHS-based consumer-choice model, show that the recovered policies closely match expert behaviour.
  arXiv  Detail & Related papers  (2026-06-15T14:13:16Z)
• Decoding Rewards in Competitive Games: Inverse Game Theory with Entropy Regularization [52.74762030521324]
  We propose a novel algorithm to learn reward functions from observed actions.<n>We provide strong theoretical guarantees for the reliability and sample efficiency of our algorithm.
  arXiv  Detail & Related papers  (2026-01-19T04:12:51Z)
• Spectral Analysis of Hard-Constraint PINNs: The Spatial Modulation Mechanism of Boundary Functions [4.170072254495455]
  This work reveals that the boundary function $B$ introduces a multiplicative spatial modulation that fundamentally alters the learning landscape.<n>A rigorous Neural Tangent Kernel (NTK) framework for HC-PINNs is established, deriving the explicit kernel composition law.<n>It is shown that widely used boundary functions can inadvertently induce spectral collapse, leading to optimization stagnation despite exact boundary satisfaction.
  arXiv  Detail & Related papers  (2025-12-29T08:31:58Z)
• Random Gradient-Free Optimization in Infinite Dimensional Spaces [3.8031924942083517]
  We propose a random gradient-free method for optimization in infinite dimensional Hilbert spaces.<n>Our framework requires only the computation of directional derivatives and a pre-basis for the Hilbert space domain.<n>We showcase the use of our method to solve partial differential equations la physics informed neural networks.
  arXiv  Detail & Related papers  (2025-12-23T18:09:49Z)
• Statistical analysis of Inverse Entropy-regularized Reinforcement Learning [15.054399128586232]
  Inverse reinforcement learning aims to infer the reward function that explains expert behavior observed through trajectories of state--action pairs.<n>Many reward functions can induce the same optimal policy, rendering the inverse problem ill-posed.<n>We develop a statistical framework for Inverse Entropy-regularized Reinforcement Learning.
  arXiv  Detail & Related papers  (2025-12-07T18:26:19Z)
• An Empirical Risk Minimization Approach for Offline Inverse RL and Dynamic Discrete Choice Model [9.531082746970286]
  We study the problem of estimating Dynamic Choice (DDC) models, also known as offline Maximum Entropy-Regularized Inverse Reinforcement Learning ( offline MaxEnt-IRL) in machine learning.<n>The objective is to recover reward or $Q*$ functions that govern agent behavior from offline behavior data.<n>We propose a globally convergent gradient-based method for solving these problems without the restrictive assumption of linearly parameterized rewards.
  arXiv  Detail & Related papers  (2025-02-19T22:22:20Z)
• Stability Bounds for the Unfolded Forward-Backward Algorithm [13.537414663819971]
  We consider a neural network architecture designed to solve inverse problems where the degradation operator is linear and known.<n> robustness of this inversion method to input perturbations is analyzed theoretically.<n>A key novelty of our work lies in examining the robustness of the proposed network to perturbations in its bias.
  arXiv  Detail & Related papers  (2024-12-23T11:55:41Z)
• Stable Nonconvex-Nonconcave Training via Linear Interpolation [51.668052890249726]
  This paper presents a theoretical analysis of linearahead as a principled method for stabilizing (large-scale) neural network training. We argue that instabilities in the optimization process are often caused by the nonmonotonicity of the loss landscape and show how linear can help by leveraging the theory of nonexpansive operators.
  arXiv  Detail & Related papers  (2023-10-20T12:45:12Z)
• Linear convergence of forward-backward accelerated algorithms without knowledge of the modulus of strong convexity [14.0409219811182]
  We show that both Nesterov's accelerated gradient descent (NAG) and FISTA exhibit linear convergence for strongly convex functions. We emphasize the distinctive approach employed in crafting the Lyapunov function, which involves a dynamically adapting coefficient of kinetic energy.
  arXiv  Detail & Related papers  (2023-06-16T08:58:40Z)
• Learning to Optimize with Stochastic Dominance Constraints [103.26714928625582]
  In this paper, we develop a simple yet efficient approach for the problem of comparing uncertain quantities. We recast inner optimization in the Lagrangian as a learning problem for surrogate approximation, which bypasses apparent intractability. The proposed light-SD demonstrates superior performance on several representative problems ranging from finance to supply chain management.
  arXiv  Detail & Related papers  (2022-11-14T21:54:31Z)
• Learning Two-Player Mixture Markov Games: Kernel Function Approximation and Correlated Equilibrium [157.0902680672422]
  We consider learning Nash equilibria in two-player zero-sum Markov Games with nonlinear function approximation. We propose a novel online learning algorithm to find a Nash equilibrium by minimizing the duality gap.
  arXiv  Detail & Related papers  (2022-08-10T14:21:54Z)
• Gradient Backpropagation Through Combinatorial Algorithms: Identity with Projection Works [20.324159725851235]
  A meaningful replacement for zero or undefined solvers is crucial for effective gradient-based learning. We propose a principled approach to exploit the geometry of the discrete solution space to treat the solver as a negative identity on the backward pass.
  arXiv  Detail & Related papers  (2022-05-30T16:17:09Z)
• Trajectory Inference via Mean-field Langevin in Path Space [0.17205106391379024]
  Trajectory inference aims at recovering the dynamics of a population from snapshots of its temporal marginals. A min-entropy estimator relative to the Wiener measure in path space was introduced by Lavenant et al.
  arXiv  Detail & Related papers  (2022-05-14T23:13:00Z)
• On the Benefits of Large Learning Rates for Kernel Methods [110.03020563291788]
  We show that a phenomenon can be precisely characterized in the context of kernel methods. We consider the minimization of a quadratic objective in a separable Hilbert space, and show that with early stopping, the choice of learning rate influences the spectral decomposition of the obtained solution.
  arXiv  Detail & Related papers  (2022-02-28T13:01:04Z)
• Anti-Concentrated Confidence Bonuses for Scalable Exploration [57.91943847134011]
  Intrinsic rewards play a central role in handling the exploration-exploitation trade-off. We introduce emphanti-concentrated confidence bounds for efficiently approximating the elliptical bonus. We develop a practical variant for deep reinforcement learning that is competitive with contemporary intrinsic rewards on Atari benchmarks.
  arXiv  Detail & Related papers  (2021-10-21T15:25:15Z)
• Lifting the Convex Conjugate in Lagrangian Relaxations: A Tractable Approach for Continuous Markov Random Fields [53.31927549039624]
  We show that a piecewise discretization preserves better contrast from existing discretization problems. We apply this theory to the problem of matching two images.
  arXiv  Detail & Related papers  (2021-07-13T12:31:06Z)
• A Stochastic Composite Augmented Lagrangian Method For Reinforcement Learning [9.204659134755795]
  We consider the linear programming (LP) formulation for deep reinforcement learning. The augmented Lagrangian method suffers the double-sampling obstacle in solving the LP. A deep parameterized augment Lagrangian method is proposed.
  arXiv  Detail & Related papers  (2021-05-20T13:08:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.