Phi-Actor-Critic: Steering General-Sum Games to Pareto-Efficient Correlated Equilibria

• URL: http://arxiv.org/abs/2606.11284v1
• Date: Tue, 09 Jun 2026 16:40:26 GMT
• Title: Phi-Actor-Critic: Steering General-Sum Games to Pareto-Efficient Correlated Equilibria
• Abstract summary: Real-world multi-agent systems are often modeled as general-sum games where individual incentives conflict with collective welfare.<n>Standard deep multi-agent reinforcement learning (MARL) methods struggle with this problem.<n>We propose $$-Actor-Critic ($$-AC), a framework that leverages swap regret minimization to steer learning toward high-welfare correlated equilibria.
• Score: 3.061219970798378
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Real-world multi-agent systems, from traffic coordination to resource allocation, are often modeled as general-sum games where individual incentives conflict with collective welfare. In these settings, the central challenge is not merely finding an equilibrium, but selecting socially desirable outcomes among many suboptimal Nash equilibria. Standard deep multi-agent reinforcement learning (MARL) methods struggle with this problem, as value-decomposition approaches are constrained by monotonicity assumptions and policy-gradient methods often converge to stable but socially inefficient equilibria. To address this limitation, we propose $Φ$-Actor-Critic ($Φ$-AC), a framework that leverages swap regret minimization to steer learning toward high-welfare correlated equilibria (CE). To make counterfactual regret estimation tractable in deep MARL, $Φ$-AC employs a centralized attention critic that predicts vector-valued regrets in a single forward pass, avoiding computationally expensive counterfactual simulations. We further introduce a Lagrangian-based equilibrium selection mechanism that optimizes social welfare while enforcing stability through regret constraints. Experiments on matrix games, Multi-Agent Particle Environments (MPE), and the Melting Pot Harvest scenario demonstrate that $Φ$-AC learns efficient and stable coordination strategies across diverse mixed-motive settings while maintaining high collective return and competitive fairness.

Related papers

• Stable and Budget-Feasible Coalition Formation for Clustered Federated Learning: A Hedonic Potential-Game Approach [0.23689955632456086]
  Clustered learning benefits from organizing heterogeneous participants into coalitions that train coalition-specific models.<n>We develop a transferable-surplus model separating learning benefit, system cost, participant cost, and monetary transfers.
  arXiv  Detail & Related papers  (2026-07-29T11:29:45Z)
• Paradoxes of Game Theoretic Equilibria and Price of Anarchy [36.44731230315158]
  We show that reducing multi-agent learning to equilibrium obscures underlying dynamic disequilibrium and gameoretic bounds.<n>Worst-case pure Nash equilibria dictating robust Price of Anarchy bounds manifest as topologically unstable strict saddles.<n> projecting learning trajectories onto the discrete simplex of correlated play systematically accommodates non-rationalizable behavior.
  arXiv  Detail & Related papers  (2026-07-13T16:09:35Z)
• DNQ: Deep Nash Q-Network for Partially Observable n-Player Games [4.004249151279306]
  We propose DNQ, a solver-in-the-loop equilibrium supervision framework for training bidding agents.<n>DNQ alternates between trajectory collection, critic-based payoff estimation, equilibrium, and policy imitation.<n>We focus on a scalable pairwise formulation that greatly reduces equilibrium-solving cost and training time compared with the exact formulation.
  arXiv  Detail & Related papers  (2026-06-04T17:58:01Z)
• Breaking $\textit{Winner-Takes-All}$: Cooperative Policy Optimization Improves Diverse LLM Reasoning [53.42577591449649]
  Group Cooperative Policy Optimization shifts the training paradigm from rollout competition to team cooperation.<n>GCPO replaces independent rollout scoring with team-level credit assignment.<n>It redistributes the collective team reward to each single rollout according to its average marginal contribution to the team.
  arXiv  Detail & Related papers  (2026-05-12T03:20:24Z)
• GRASP: Gradient Realignment via Active Shared Perception for Multi-Agent Collaborative Optimization [41.52202306408042]
  Non-stationarity arises from concurrent policy updates and leads to persistent environmental fluctuations.<n>We propose Realignment via Active Shared Perception (GRASP), a novel framework that defines generalized Bellman equilibrium as a stable objective for policy evolution.
  arXiv  Detail & Related papers  (2026-04-01T10:26:22Z)
• Socially-Weighted Alignment: A Game-Theoretic Framework for Multi-Agent LLM Systems [17.658093330392052]
  We propose a game-theoretic framework that modifies inference-time decision making by interpolating between an agent's private objective and an estimate of group welfare.<n>We show that SWA induces a critical threshold $*=(n-)/(n-1)$ above which agents no longer have marginal incentive to increase demand under overload.
  arXiv  Detail & Related papers  (2026-02-16T05:17:58Z)
• Integrating upstream and downstream reciprocity stabilizes cooperator-defector coexistence in N-player giving games [1.1381558444077822]
  We show how pay-it-forward chains and reputation systems can jointly maintain social including cooperation despite cognitive limitations and group size challenges.<n>This framework demonstrates how pay-it-forward chains and reputation systems can jointly maintain social including cooperation despite cognitive limitations and group size challenges.
  arXiv  Detail & Related papers  (2025-09-05T01:49:26Z)
• From Debate to Equilibrium: Belief-Driven Multi-Agent LLM Reasoning via Bayesian Nash Equilibrium [52.28048367430481]
  Multi-agent frameworks can boost the reasoning power of large language models (LLMs), but they typically incur heavy computational costs and lack convergence guarantees.<n>We recast multi-LLM coordination as an incomplete-information game and seek a Bayesian Nash equilibrium (BNE)<n>We introduce Efficient Coordination via Nash Equilibrium (ECON), a hierarchical reinforcement-learning paradigm that marries distributed reasoning with centralized final output.
  arXiv  Detail & Related papers  (2025-06-09T23:49:14Z)
• Vairiational Stochastic Games [1.6703448188585752]
  We propose a novel variational inference framework tailored to decentralized multi-agent systems.<n>Our framework addresses the challenges posed by non-stationarity and unaligned agent objectives.<n>We demonstrate theoretical convergence guarantees for the proposed decentralized algorithms.
  arXiv  Detail & Related papers  (2025-03-08T03:21:23Z)
• On Tractable $Φ$-Equilibria in Non-Concave Games [53.212133025684224]
  We study tractable $Phi$-equilibria in non-concave games.<n>We show that when $Phi$ is finite, there exists an efficient uncoupled learning algorithm that converges to the corresponding $Phi$-equilibria.
  arXiv  Detail & Related papers  (2024-03-13T01:51:30Z)
• A Black-box Approach for Non-stationary Multi-agent Reinforcement Learning [53.83345471268163]
  We investigate learning the equilibria in non-stationary multi-agent systems. We show how to test for various types of equilibria by a black-box reduction to single-agent learning.
  arXiv  Detail & Related papers  (2023-06-12T23:48:24Z)
• Inducing Equilibria via Incentives: Simultaneous Design-and-Play Finds Global Optima [114.31577038081026]
  We propose an efficient method that tackles the designer's and agents' problems simultaneously in a single loop. Although the designer does not solve the equilibrium problem repeatedly, it can anticipate the overall influence of the incentives on the agents. We prove that the algorithm converges to the global optima at a sublinear rate for a broad class of games.
  arXiv  Detail & Related papers  (2021-10-04T06:53:59Z)
• Multi-Stage Decentralized Matching Markets: Uncertain Preferences and Strategic Behaviors [91.3755431537592]
  This article develops a framework for learning optimal strategies in real-world matching markets. We show that there exists a welfare-versus-fairness trade-off that is characterized by the uncertainty level of acceptance. We prove that participants can be better off with multi-stage matching compared to single-stage matching.
  arXiv  Detail & Related papers  (2021-02-13T19:25:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.