Mechanism Design Is Not Enough: Prosocial Agents for Cooperative AI

• URL: http://arxiv.org/abs/2605.08426v1
• Date: Fri, 08 May 2026 19:37:07 GMT
• Title: Mechanism Design Is Not Enough: Prosocial Agents for Cooperative AI
• Abstract summary: We show that when contracts cannot distinguish all relevant future contingencies, there is a strictly positive welfare loss that no realistic mechanism can eliminate.<n>We show that prosocial agents, who weigh others' welfare alongside their own, can close this gap and achieve outcomes that are socially superior and individually beneficial.
• Score: 49.05972944310475
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Ensuring that AI agents behave safely and beneficially when interacting with other parties has emerged as one of the central challenges of modern AI safety. While mechanism design, as the theory of designing rules to align individual and collective objectives, can incentivize cooperative behavior, it is still an open question whether it alone is sufficient to maximize LLM agents' social welfare. This work proves that the answer is negative: drawing from incomplete contract theory, we formally show that when contracts cannot distinguish all relevant future contingencies, there is a strictly positive welfare loss that no realistic mechanism can eliminate. We show that prosocial agents, who weigh others' welfare alongside their own, can close this gap and achieve outcomes that are socially superior and individually beneficial. Experimentally, we show that in multi-agent resource-allocation environments and canonical social dilemmas where agents are powered by large language models, prosociality is beneficial. The implication for AI safety is clear: to enable cooperative interactions at scale, designing adequate mechanisms is not sufficient; agents must be built to be intrinsically prosocial.

Related papers

• A game theory for foundation models shows new paths to rational cooperation through similarity inference [56.53805518668248]
  We show that foundation model agents engaging in optimal planning consistently converge to stable cooperation.<n>By shifting from decoupled to embedded agency, these agents model themselves as part of the universe they inhabit.
  arXiv  Detail & Related papers  (2026-08-04T17:24:33Z)
• Solipsistic Superintelligence is Unlikely to be Cooperative [48.65947679109806]
  We argue that superintelligence, an extremely capable task solver, is unlikely to be cooperative.<n>We call for a non-solipsistic research paradigm that treats interdependence as a core design principle.
  arXiv  Detail & Related papers  (2026-06-02T06:54:55Z)
• Hybrid Human-Agent Social Dilemmas in Energy Markets [0.45880283710344055]
  We study how cooperative behaviors can emerge in hybrid populations where humans delegate strategic decision-making to autonomous agents.<n>We introduce artificial agents that use globally observable signals to increase coordination.<n>We show that artificial agents can shift the learning dynamics to favour coordination outcomes.
  arXiv  Detail & Related papers  (2026-03-12T11:59:35Z)
• The Devil Behind Moltbook: Anthropic Safety is Always Vanishing in Self-Evolving AI Societies [57.387081435669835]
  Multi-agent systems built from large language models offer a promising paradigm for scalable collective intelligence and self-evolution.<n>We show that an agent society satisfying continuous self-evolution, complete isolation, and safety invariance is impossible.<n>We propose several solution directions to alleviate the identified safety concern.
  arXiv  Detail & Related papers  (2026-02-10T15:18:19Z)
• Do LLMs trust AI regulation? Emerging behaviour of game-theoretic LLM agents [61.132523071109354]
  This paper investigates the interplay between AI developers, regulators and users, modelling their strategic choices under different regulatory scenarios.<n>Our research identifies emerging behaviours of strategic AI agents, which tend to adopt more "pessimistic" stances than pure game-theoretic agents.
  arXiv  Detail & Related papers  (2025-04-11T15:41:21Z)
• Cooperate or Collapse: Emergence of Sustainable Cooperation in a Society of LLM Agents [101.17919953243107]
  GovSim is a generative simulation platform designed to study strategic interactions and cooperative decision-making in large language models (LLMs)<n>We find that all but the most powerful LLM agents fail to achieve a sustainable equilibrium in GovSim, with the highest survival rate below 54%.<n>We show that agents that leverage "Universalization"-based reasoning, a theory of moral thinking, are able to achieve significantly better sustainability.
  arXiv  Detail & Related papers  (2024-04-25T15:59:16Z)
• Mediated Multi-Agent Reinforcement Learning [3.8581550679584473]
  We show how a mediator can be trained alongside agents with policy gradient to maximize social welfare. Our experiments in matrix and iterative games highlight the potential power of applying mediators in Multi-Agent Reinforcement Learning.
  arXiv  Detail & Related papers  (2023-06-14T10:31:37Z)
• Towards a Unifying Model of Rationality in Multiagent Systems [11.321217099465196]
  Multiagent systems need to cooperate with other agents (including humans) nearly as effectively as these agents cooperate with one another. We propose a generic model of socially intelligent agents, which are individually rational learners that are also able to cooperate with one another. We show how we can construct socially intelligent agents for different forms of regret.
  arXiv  Detail & Related papers  (2023-05-29T13:18:43Z)
• The art of compensation: how hybrid teams solve collective risk dilemmas [6.081979963786028]
  We study the evolutionary dynamics of cooperation in a hybrid population made of both adaptive and fixed-behavior agents. We show how the first learn to adapt their behavior to compensate for the behavior of the latter.
  arXiv  Detail & Related papers  (2022-05-13T13:23:42Z)
• Cooperation and Reputation Dynamics with Reinforcement Learning [6.219565750197311]
  We show how reputations can be used as a way to establish trust and cooperation. We propose two mechanisms to alleviate convergence to undesirable equilibria. We show how our results relate to the literature in Evolutionary Game Theory.
  arXiv  Detail & Related papers  (2021-02-15T12:48:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.