Equilibrium Selection in Multi-Agent Policy Gradients via Opponent-Aware Basin Entry

• URL: http://arxiv.org/abs/2605.18078v1
• Date: Mon, 18 May 2026 08:57:15 GMT
• Title: Equilibrium Selection in Multi-Agent Policy Gradients via Opponent-Aware Basin Entry
• Abstract summary: We study basin-entry probability with respect to a target set of equilibria selected by an external criterion.<n>For finite-unroll Meta-MAPG, we show that the update decomposes into ordinary policy gradient plus own-learning and peer-learning corrections.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multi-agent policy-gradient methods have been shown to converge locally near stable Nash equilibria. Local convergence, however, does not determine which equilibrium is reached. We study this question through basin-entry probability with respect to a target set of equilibria selected by an external criterion, such as payoff dominance. For finite-unroll Meta-MAPG, we show that the update decomposes into ordinary policy gradient plus own-learning and peer-learning corrections, with controlled sampling noise and finite-unroll bias. We identify the peer-learning correction as the main equilibrium-selection mechanism: under a local alignment condition, the probability of entering the certified attraction region of the target stable-Nash set increases, relative to ordinary policy gradient. Because persistent correction may shift zero-update points of the original game, annealing the correction after entering the basin recovers ordinary policy-gradient dynamics and inherits local stable-Nash convergence guarantees. Experiments in Stag Hunt, iterated Prisoner's Dilemma, and preliminary neural-policy coordination environments support this basin-entry view, showing increased entry into cooperative basins under peer-aware updates.

Related papers

• Convex-Hull-Neighborhood Smooth Dual Generalization: Controlling Local Correction Propagation in Offline RL [7.803284875836737]
  bootstrapping can amplify estimation errors in offline reinforcement learning.<n>We propose Convex Hull Neighborhood Smooth Dual Generalization, which expresses the Bellman backup as an in-sample value target plus a CHN-local correction.<n> Experiments on Gym-MuJoCo and AntMaze show strong aggregate performance and stable value estimation.
  arXiv  Detail & Related papers  (2026-08-04T04:30:02Z)
• Phi-Actor-Critic: Steering General-Sum Games to Pareto-Efficient Correlated Equilibria [3.061219970798378]
  Real-world multi-agent systems are often modeled as general-sum games where individual incentives conflict with collective welfare.<n>Standard deep multi-agent reinforcement learning (MARL) methods struggle with this problem.<n>We propose $$-Actor-Critic ($$-AC), a framework that leverages swap regret minimization to steer learning toward high-welfare correlated equilibria.
  arXiv  Detail & Related papers  (2026-06-09T16:40:26Z)
• GRASP: Gradient Realignment via Active Shared Perception for Multi-Agent Collaborative Optimization [41.52202306408042]
  Non-stationarity arises from concurrent policy updates and leads to persistent environmental fluctuations.<n>We propose Realignment via Active Shared Perception (GRASP), a novel framework that defines generalized Bellman equilibrium as a stable objective for policy evolution.
  arXiv  Detail & Related papers  (2026-04-01T10:26:22Z)
• Improving Search Agent with One Line of Code [68.58667107354253]
  Tool-based Agentic Reinforcement Learning (TARL) has emerged as a promising paradigm for training search agents to interact with external tools.<n>We propose textbfSearch textbfAgent textbfPolicy textbfOptimization (textbfSAPO), which stabilizes training via a conditional token-level KL constraint.
  arXiv  Detail & Related papers  (2026-03-10T04:07:39Z)
• Asymmetric regularization mechanism for GAN training with Variational Inequalities [1.529943343419486]
  We formulate the training of generative adversarial networks (GANs) as a Nash equilibrium seeking problem.<n>We propose an asymmetric regularization mechanism based on the classic Tikhonov step and on a novel zero-centered gradient penalty.
  arXiv  Detail & Related papers  (2026-01-20T12:50:18Z)
• Entropy Ratio Clipping as a Soft Global Constraint for Stable Reinforcement Learning [49.92803982100042]
  We propose using the entropy ratio between the current and previous policies as a new global metric.<n>We introduce an textbfEntropy Ratio (ERC) mechanism that imposes bidirectional constraints on the entropy ratio.<n>This stabilizes policy updates at the global distribution level and compensates for the inability of PPO-clip to regulate probability shifts of un-sampled actions.
  arXiv  Detail & Related papers  (2025-12-05T10:26:32Z)
• Nash Policy Gradient: A Policy Gradient Method with Iteratively Refined Regularization for Finding Nash Equilibria [27.756691720415798]
  We develop a practical algorithm for finding Nash equilibria in imperfect-information games.<n>Nash Policy Gradient (NashPG) achieves comparable or lower exploitability than prior model-free methods on classic benchmark games.
  arXiv  Detail & Related papers  (2025-10-21T00:14:45Z)
• Towards Optimal Offline Reinforcement Learning [9.13232872223434]
  We study offline reinforcement learning problems with a long-run average reward objective.<n>The state-action pairs generated by any fixed behavioral policy thus follow a Markov chain.<n>We use the rate function of this large deviations principle to construct an uncertainty set for the unknown em true state-action-next-state distribution.
  arXiv  Detail & Related papers  (2025-03-15T22:41:55Z)
• Gradient Equilibrium in Online Learning: Theory and Applications [56.02856551198923]
  gradient equilibrium is achieved by standard online learning methods.<n> gradient equilibrium translates into an interpretable and meaningful property in online prediction problems.<n>We show that gradient equilibrium framework can be used to develop a debiasing scheme for black-box predictions.
  arXiv  Detail & Related papers  (2025-01-14T18:59:09Z)
• The Role of Baselines in Policy Gradient Optimization [83.42050606055822]
  We show that the emphstate value baseline allows on-policy. emphnatural policy gradient (NPG) to converge to a globally optimal. policy at an $O (1/t) rate gradient. We find that the primary effect of the value baseline is to textbfreduce the aggressiveness of the updates rather than their variance.
  arXiv  Detail & Related papers  (2023-01-16T06:28:00Z)
• A comment on stabilizing reinforcement learning [0.0]
  We argue that Vamvoudakis et al. made a fallacious assumption on the Hamiltonian under a generic policy. We show a neural network convergence under a continuous-weight-time environment, provided certain conditions on the behavior policy hold.
  arXiv  Detail & Related papers  (2021-11-24T07:58:14Z)
• Dealing with Non-Stationarity in Multi-Agent Reinforcement Learning via Trust Region Decomposition [52.06086375833474]
  Non-stationarity is one thorny issue in multi-agent reinforcement learning. We introduce a $delta$-stationarity measurement to explicitly model the stationarity of a policy sequence. We propose a trust region decomposition network based on message passing to estimate the joint policy divergence.
  arXiv  Detail & Related papers  (2021-02-21T14:46:50Z)
• Improper Learning with Gradient-based Policy Optimization [62.50997487685586]
  We consider an improper reinforcement learning setting where the learner is given M base controllers for an unknown Markov Decision Process. We propose a gradient-based approach that operates over a class of improper mixtures of the controllers.
  arXiv  Detail & Related papers  (2021-02-16T14:53:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.