Towards Model-Free Learning in Dynamic Population Games: An Application to Karma Economies

• URL: http://arxiv.org/abs/2605.11042v1
• Date: Mon, 11 May 2026 08:39:30 GMT
• Title: Towards Model-Free Learning in Dynamic Population Games: An Application to Karma Economies
• Abstract summary: Existing computational tools for Dynamic Population Games assume full knowledge of the game model and operate in a centralized fashion.<n>This paper takes a step towards addressing this gap by studying model-free equilibrium learning in Karma DPGs.<n>We show that combining deep RL with fictitious play and smoothed policy allows agents to converge, in a model-free fashion, to a configuration close to the centrally computed SNE.
• Score: 7.726701007187284
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Dynamic Population Games (DPGs) provide a tractable framework for modeling strategic interactions in large populations of self-interested agents, and have been successfully applied to the design of Karma economies, a class of fair non-monetary resource allocation mechanisms. Despite their appealing theoretical properties, existing computational tools for DPGs assume full knowledge of the game model and operate in a centralized fashion, limiting their applicability in realistic settings where agents have access only to their own private experience. This paper takes a step towards addressing this gap by studying model-free equilibrium learning in Karma DPGs. First, we analyze the setting in which a novel agent joins a Karma DPG already at its Stationary Nash Equilibrium (SNE) and learns a policy via Deep Q-Networks (DQN) without knowledge of the game model. Leveraging recent convergence results for DQN, we establish a suboptimality bound consisting of a DQN approximation error of order $O(1/\sqrt{N_s})$ and a mean field perturbation error of order $O(1/N)$, where $N_s$ is the replay buffer size and $N$ is the population size. Second, we consider the challenging problem of learning the SNE from scratch. We show empirically that combining deep RL with fictitious play and smoothed policy iteration allows agents to converge, in a model-free fashion, to a configuration close to the centrally computed SNE. Together, these contributions support the vision of Karma economies as practical tools for fair resource allocation.

Related papers

• Inverting the Bellman Equation: From $Q$-Values to World Models [57.827849584133425]
  We prove that value-based agents trained on a sufficiently rich set of reward functions implicitly encode a unique and accurate world model.<n>We find that policies trained exclusively on a ttReacher agent's implicit world model are quasi-optimal on out-of-distribution, velocity-based goals despite position-only training.
  arXiv  Detail & Related papers  (2026-06-19T07:26:14Z)
• OpenVLThinkerV2: A Generalist Multimodal Reasoning Model for Multi-domain Visual Tasks [81.07598709704628]
  We present a novel RL training objective that replaces standard linear scaling with non-linear distributional matching.<n>We also present OpenVLThinkerV2, a highly robust, general-purpose multimodal model.
  arXiv  Detail & Related papers  (2026-04-09T17:59:39Z)
• Deep Hierarchical Learning with Nested Subspace Networks [53.71337604556311]
  We propose Nested Subspace Networks (NSNs) for large neural networks.<n>NSNs enable a single model to be dynamically and granularly adjusted across a continuous spectrum of compute budgets.<n>We show that NSNs can be surgically applied to pre-trained LLMs and unlock a smooth and predictable compute-performance frontier.
  arXiv  Detail & Related papers  (2025-09-22T15:13:14Z)
• Intention-Conditioned Flow Occupancy Models [80.42634994902858]
  Large-scale pre-training has fundamentally changed how machine learning research is done today.<n>Applying this same framework to reinforcement learning is appealing because it offers compelling avenues for addressing core challenges in RL.<n>Recent advances in generative AI have provided new tools for modeling highly complex distributions.
  arXiv  Detail & Related papers  (2025-06-10T15:27:46Z)
• Diffusion Self-Weighted Guidance for Offline Reinforcement Learning [1.7614751781649955]
  offline reinforcement learning (RL) recovers the optimal policy $pi$ given historical observations of an agent.<n>In practice, $pi$ is modeled as a weighted version of the agent's behavior policy $mu$, using a weight function $w$ working as a critic of the agent's behavior.<n>We show that Self-Weighted Guidance (SWG) generates samples from the desired distribution on toy examples and performs on par with state-of-the-art methods on D4RL's challenging environments.
  arXiv  Detail & Related papers  (2025-05-23T20:03:36Z)
• Jointly Training and Pruning CNNs via Learnable Agent Guidance and Alignment [69.33930972652594]
  We propose a novel structural pruning approach to jointly learn the weights and structurally prune architectures of CNN models. The core element of our method is a Reinforcement Learning (RL) agent whose actions determine the pruning ratios of the CNN model's layers. We conduct the joint training and pruning by iteratively training the model's weights and the agent's policy.
  arXiv  Detail & Related papers  (2024-03-28T15:22:29Z)
• On Tractable $Φ$-Equilibria in Non-Concave Games [53.212133025684224]
  We study tractable $Phi$-equilibria in non-concave games.<n>We show that when $Phi$ is finite, there exists an efficient uncoupled learning algorithm that converges to the corresponding $Phi$-equilibria.
  arXiv  Detail & Related papers  (2024-03-13T01:51:30Z)
• Population-aware Online Mirror Descent for Mean-Field Games by Deep Reinforcement Learning [43.004209289015975]
  Mean Field Games (MFGs) have the ability to handle large-scale multi-agent systems. We propose a deep reinforcement learning (DRL) algorithm that achieves population-dependent Nash equilibrium.
  arXiv  Detail & Related papers  (2024-03-06T08:55:34Z)
• Efficient Model-Based Multi-Agent Mean-Field Reinforcement Learning [89.31889875864599]
  We propose an efficient model-based reinforcement learning algorithm for learning in multi-agent systems. Our main theoretical contributions are the first general regret bounds for model-based reinforcement learning for MFC. We provide a practical parametrization of the core optimization problem.
  arXiv  Detail & Related papers  (2021-07-08T18:01:02Z)
• Reinforcement Learning for Mean Field Games, with Applications to Economics [0.0]
  Mean field games (MFG) and mean field control problems (MFC) are frameworks to study Nash equilibria or social optima in games with a continuum of agents. We present a two timescale approach with RL for MFG and MFC, which relies on a unified Q-learning algorithm.
  arXiv  Detail & Related papers  (2021-06-25T16:45:04Z)
• Operator Splitting for Learning to Predict Equilibria in Convex Games [26.92001486095397]
  We introduce Nash Fixed Point Networks (N-FPNs), a class of neural networks that naturally output equilibria. N-FPNs are compatible with the recently developed Jacobian-Free Backpropagation technique for training implicit networks. Our experiments show N-FPNs are capable of scaling to problems orders of magnitude larger than existing learned game solvers.
  arXiv  Detail & Related papers  (2021-06-02T02:55:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.