The Price of Paranoia: Robust Risk-Sensitive Cooperation in Non-Stationary Multi-Agent Reinforcement Learning

• URL: http://arxiv.org/abs/2604.15695v1
• Date: Fri, 17 Apr 2026 04:41:38 GMT
• Title: The Price of Paranoia: Robust Risk-Sensitive Cooperation in Non-Stationary Multi-Agent Reinforcement Learning
• Abstract summary: We study how co-learning noise propagates through the structure of coordination games.<n>We find that the cooperative equilibrium is exponentially unstable under standard risk-neutral learning.<n>We show that robustness should target the policy gradient update variance induced by partner uncertainty, not the return distribution.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Cooperative equilibria are fragile. When agents learn alongside each other rather than in a fixed environment, the process of learning destabilizes the cooperation they are trying to sustain: every gradient step an agent takes shifts the distribution of actions its partner will play, turning a cooperative partner into a source of stochastic noise precisely where the cooperation decision is most sensitive. We study how this co-learning noise propagates through the structure of coordination games, and find that the cooperative equilibrium, even when strongly Pareto-dominant, is exponentially unstable under standard risk-neutral learning, collapsing irreversibly once partner noise crosses the game's critical cooperation threshold. The natural response to apply distributional robustness to hedge against partner uncertainty makes things strictly worse: risk-averse return objectives penalize the high-variance cooperative action relative to defection, widening the instability region rather than shrinking it, a paradox that reveals a fundamental mismatch between the domains where robustness is applied and instability originates. We resolve this by showing that robustness should target the policy gradient update variance induced by partner uncertainty, not the return distribution. This distinction yields an algorithm whose gradient updates are modulated by an online measure of partner unpredictability, provably expanding the cooperation basin in symmetric coordination games. To unify stability, sample complexity, and welfare consequences of this approach, we introduce the Price of Paranoia as the structural dual of the Price of Anarchy. Together with a novel Cooperation Window, it precisely characterizes how much welfare learning algorithms can recover under partner noise, pinning down the optimal degree of robustness as a closed-form balance between equilibrium stability and sample efficiency.

Related papers

• Exit-and-Join Dynamics and Equilibrium in Continuum Cooperative Games [20.036142577511455]
  We develop a continuum theory of exit-and-join coalition dynamics in nonatomic cooperative games.<n>We derive mean-field dynamics from decentralized switching rules and show that payoff-difference switching recovers replicator dynamics.<n>The proposed theory unifies cooperative value allocation, noncooperative coalition mobility, mean-field dynamics, evolutionary game theory, and population games.
  arXiv  Detail & Related papers  (2026-06-27T09:17:07Z)
• Stationary Robust Mean-Field Games under Model Mismatches [3.053409604085541]
  This paper develops an infinite-horizon, stationary mean-field game framework.<n>It incorporates distributional model uncertainty directly into the population-coupled dynamics.<n>Under a contractive robust-dynamics regime, we obtain explicit non-asymptotic error bounds.
  arXiv  Detail & Related papers  (2026-06-21T16:29:25Z)
• Adaptive Smooth Tchebycheff Attention for Multi-Objective Policy Optimization [55.338784037660474]
  Multi- curvature reinforcement learning in domains requires balancing complex, non-fragile interference trade-offs between conflicting objectives.<n>In this paper we introduce a novel based real-time based real-time optimization on a robotic visual -- a proxy for monitoring ecosystems.
  arXiv  Detail & Related papers  (2026-05-12T21:32:57Z)
• Robust Adversarial Policy Optimization Under Dynamics Uncertainty [8.729531978655737]
  Reinforcement learning (RL) policies often fail under dynamics that differ from training.<n>We propose a dual formulation that directly exposes the robustness-performance trade-off.<n>The resulting framework, robust adversarial policy optimization (RAPO), outperforms robust RL baselines.
  arXiv  Detail & Related papers  (2026-04-13T04:23:54Z)
• GRASP: Gradient Realignment via Active Shared Perception for Multi-Agent Collaborative Optimization [41.52202306408042]
  Non-stationarity arises from concurrent policy updates and leads to persistent environmental fluctuations.<n>We propose Realignment via Active Shared Perception (GRASP), a novel framework that defines generalized Bellman equilibrium as a stable objective for policy evolution.
  arXiv  Detail & Related papers  (2026-04-01T10:26:22Z)
• Decentralized Dynamic Cooperation of Personalized Models for Federated Continual Learning [50.56947843548702]
  We propose a decentralized dynamic cooperation framework for Federated continual learning.<n>Clients establish dynamic cooperative learning coalitions to balance the acquisition of new knowledge and the retention of prior learning.<n>We also propose a merge-blocking algorithm and a dynamic cooperative evolution algorithm to achieve cooperative and dynamic equilibrium.
  arXiv  Detail & Related papers  (2025-09-28T06:53:23Z)
• Integrating upstream and downstream reciprocity stabilizes cooperator-defector coexistence in N-player giving games [1.1381558444077822]
  We show how pay-it-forward chains and reputation systems can jointly maintain social including cooperation despite cognitive limitations and group size challenges.<n>This framework demonstrates how pay-it-forward chains and reputation systems can jointly maintain social including cooperation despite cognitive limitations and group size challenges.
  arXiv  Detail & Related papers  (2025-09-05T01:49:26Z)
• CooT: Learning to Coordinate In-Context with Coordination Transformers [10.888155149916967]
  Coordination Transformers (coot) is a novel in-context coordination framework that rapidly adapts to unseen partners.<n>coot consistently outperforms baselines including population-based approaches, gradient-based fine-tuning, and a Meta-RL-inspired contextual adaptation method.<n>By contrast, coot achieves stable, rapid in-context adaptation and is consistently ranked the most effective collaborator in human evaluations.
  arXiv  Detail & Related papers  (2025-06-30T06:45:39Z)
• Seeing is not Believing: Robust Reinforcement Learning against Spurious Correlation [57.351098530477124]
  We consider one critical type of robustness against spurious correlation, where different portions of the state do not have correlations induced by unobserved confounders. A model that learns such useless or even harmful correlation could catastrophically fail when the confounder in the test case deviates from the training one. Existing robust algorithms that assume simple and unstructured uncertainty sets are therefore inadequate to address this challenge.
  arXiv  Detail & Related papers  (2023-07-15T23:53:37Z)
• Distributed Adaptive Learning Under Communication Constraints [54.22472738551687]
  This work examines adaptive distributed learning strategies designed to operate under communication constraints. We consider a network of agents that must solve an online optimization problem from continual observation of streaming data.
  arXiv  Detail & Related papers  (2021-12-03T19:23:48Z)
• Learning Equilibria in Matching Markets from Bandit Feedback [139.29934476625488]
  We develop a framework and algorithms for learning stable market outcomes under uncertainty. Our work takes a first step toward elucidating when and how stable matchings arise in large, data-driven marketplaces.
  arXiv  Detail & Related papers  (2021-08-19T17:59:28Z)
• Balancing Rational and Other-Regarding Preferences in Cooperative-Competitive Environments [4.705291741591329]
  Mixed environments are notorious for the conflicts of selfish and social interests. We propose BAROCCO to balance individual and social incentives. Our meta-algorithm is compatible with both Q-learning and Actor-Critic frameworks.
  arXiv  Detail & Related papers  (2021-02-24T14:35:32Z)
• Dealing with Non-Stationarity in Multi-Agent Reinforcement Learning via Trust Region Decomposition [52.06086375833474]
  Non-stationarity is one thorny issue in multi-agent reinforcement learning. We introduce a $delta$-stationarity measurement to explicitly model the stationarity of a policy sequence. We propose a trust region decomposition network based on message passing to estimate the joint policy divergence.
  arXiv  Detail & Related papers  (2021-02-21T14:46:50Z)
• Implicit Distributional Reinforcement Learning [61.166030238490634]
  implicit distributional actor-critic (IDAC) built on two deep generator networks (DGNs) Semi-implicit actor (SIA) powered by a flexible policy distribution. We observe IDAC outperforms state-of-the-art algorithms on representative OpenAI Gym environments.
  arXiv  Detail & Related papers  (2020-07-13T02:52:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.