Institutional AI: Governing LLM Collusion in Multi-Agent Cournot Markets via Public Governance Graphs

• URL: http://arxiv.org/abs/2601.11369v2
• Date: Tue, 20 Jan 2026 12:10:21 GMT
• Title: Institutional AI: Governing LLM Collusion in Multi-Agent Cournot Markets via Public Governance Graphs
• Abstract summary: This paper advances an experimental framework for evaluating Institutional AI.<n>Central to this approach is the governance graph, a public, immutable manifest that declares legal states, transitions, sanctions, and restorative paths.<n>We compare three regimes: Ungoverned (baseline incentives from the structure of the Cournot market), Constitutional (a prompt-only policy-as-prompt prohibition implemented as a fixed written anti-collusion constitution) and Institutional.
• Score: 1.3763052684269788
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multi-agent LLM ensembles can converge on coordinated, socially harmful equilibria. This paper advances an experimental framework for evaluating Institutional AI, our system-level approach to AI alignment that reframes alignment from preference engineering in agent-space to mechanism design in institution-space. Central to this approach is the governance graph, a public, immutable manifest that declares legal states, transitions, sanctions, and restorative paths; an Oracle/Controller runtime interprets this manifest, attaching enforceable consequences to evidence of coordination while recording a cryptographically keyed, append-only governance log for audit and provenance. We apply the Institutional AI framework to govern the Cournot collusion case documented by prior work and compare three regimes: Ungoverned (baseline incentives from the structure of the Cournot market), Constitutional (a prompt-only policy-as-prompt prohibition implemented as a fixed written anti-collusion constitution, and Institutional (governance-graph-based). Across six model configurations including cross-provider pairs (N=90 runs/condition), the Institutional regime produces large reductions in collusion: mean tier falls from 3.1 to 1.8 (Cohen's d=1.28), and severe-collusion incidence drops from 50% to 5.6%. The prompt-only Constitutional baseline yields no reliable improvement, illustrating that declarative prohibitions do not bind under optimisation pressure. These results suggest that multi-agent alignment may benefit from being framed as an institutional design problem, where governance graphs can provide a tractable abstraction for alignment-relevant collective behavior.

Related papers

• Grounded Normative Rule Generation with Structured Search [47.28595196084396]
  We formalize the problem as Grounded Normative Rule Synthesis (GNRS)<n>We introduce GNRS-Search, a framework that utilizes Markov Chain Monte Carlo (MCMC) sampling to optimize a discrete, five-slot And-Or Graph (AOG)<n>We evaluate our approach on GNRS-Bench, a benchmark spanning 116 controlled goals across eight scene families, and RealCharterBench, which evaluates transfer to 53 real-derived policy tasks with hidden source clauses.
  arXiv  Detail & Related papers  (2026-08-23T05:43:42Z)
• Norm Enforcement for AI Agents: Robustly Shaping Behavior in Multi-Agent Systems [41.0957339989053]
  We study norm enforcement mechanisms for language model agents.<n>We find that simple enforcement mechanisms are exploited by misaligned agents for competitive advantage.<n>Our results position norm enforcement mechanisms as scalable levers for shaping agents' behavior, but only when designed to anticipate becoming part of the system they govern.
  arXiv  Detail & Related papers  (2026-07-07T06:40:14Z)
• Citation-Closure Retrieval and Per-Rule Attribution for Real-World Regulatory Compliance Question Answering [54.97384993676565]
  We formalize Regulatory Compliance QA with RegOps-Bench, a novel benchmark featuring an Operational Knowledge Graph derived from complex national R&D regulations.<n>RefWalk traverses cross-document citations, fuses multi-view candidates via max-based aggregation, and enforces per-rule attribution to explicitly map claims to sources.
  arXiv  Detail & Related papers  (2026-05-28T10:38:38Z)
• Who judges the judges? Governance from metrics: a runtime framework for continuous LLM compliance monitoring [0.0]
  Current approaches to AI compliance treat conformity as a binary, audit-time verdict.<n>We argue that this compliance fiction is structurally ill-suited to the requirements of the EU AI Act.<n>We introduce governance from metrics, a principle whereby regulatory compliance is derived as a continuous signal from behavioural observability.
  arXiv  Detail & Related papers  (2026-05-23T21:21:33Z)
• When Agents Evolve, Institutions Follow [52.359340707289114]
  We argue that multi-agent systems built on large language models face the same challenge.<n>Their central problem is not only individual intelligence, but collective organization.<n>We translate seven historical political institutions, spanning four canonical governance patterns, into executable multi-agent architectures.
  arXiv  Detail & Related papers  (2026-04-30T10:30:58Z)
• PolicyBank: Evolving Policy Understanding for LLM Agents [51.86716874651299]
  PolicyBank maintains structured, tool-level policy insights and iteratively refines them.<n>PolicyBank closes up to 82% of the gap toward a human oracle.
  arXiv  Detail & Related papers  (2026-04-16T20:29:30Z)
• From Disclosure to Self-Referential Opacity: Six Dimensions of Strain in Current AI Governance [0.0]
  This paper applies a six-dimension framework from political theory to six AI governance arrangements already in operation.<n>The patterns are offered as hypotheses for multi-rater validation.
  arXiv  Detail & Related papers  (2026-04-15T16:39:28Z)
• Governed Reasoning for Institutional AI [0.0]
  Cognitive Core is a governed decision substrate built from nine typed cognitive primitives.<n>Human review is a condition of execution rather than a post-hoc check.<n>We benchmark three systems on an 11-case balanced prior authorization appeal evaluation set.
  arXiv  Detail & Related papers  (2026-04-12T14:09:18Z)
• GRASP: Gradient Realignment via Active Shared Perception for Multi-Agent Collaborative Optimization [41.52202306408042]
  Non-stationarity arises from concurrent policy updates and leads to persistent environmental fluctuations.<n>We propose Realignment via Active Shared Perception (GRASP), a novel framework that defines generalized Bellman equilibrium as a stable objective for policy evolution.
  arXiv  Detail & Related papers  (2026-04-01T10:26:22Z)
• I Can't Believe It's Corrupt: Evaluating Corruption in Multi-Agent Governance Systems [8.670200461690454]
  We present evidence that integrity in institutional AI should be treated as a pre-deployment requirement rather than a post-deployment assumption.<n>We evaluate multi-agent governance simulations in which agents occupy formal governmental roles under different authority structures.<n>We score rule-breaking and abuse outcomes with an independent-based judge across 28,112 transcript segments.
  arXiv  Detail & Related papers  (2026-03-19T13:34:54Z)
• Incentive Aware AI Regulations: A Credal Characterisation [14.228416693145649]
  High-stakes ML applications demand strict regulations, but strategic ML providers often evade them to lower development costs.<n>We introduce regulation mechanisms: a framework that maps empirical evidence from models to a license for some market share.<n>We prove that a mechanism has perfect market outcome if and only if the set of non-compliant distributions forms a credal set of probability measures.
  arXiv  Detail & Related papers  (2026-03-05T13:42:19Z)
• Towards a Science of Collective AI: LLM-based Multi-Agent Systems Need a Transition from Blind Trial-and-Error to Rigorous Science [70.3658845234978]
  Large Language Models (LLMs) have greatly extended the capabilities of Multi-Agent Systems (MAS)<n>Despite this rapid progress, the field still relies heavily on empirical trial-and-error.<n>This bottleneck stems from the ambiguity of attribution.<n>We propose a factor attribution paradigm to systematically identify collaboration-driving factors.
  arXiv  Detail & Related papers  (2026-02-05T04:19:52Z)
• Self-Evolving Coordination Protocol in Multi-Agent AI Systems: An Exploratory Systems Feasibility Study [0.0]
  Self-Evolving Coordination Protocols (SECP)<n>SECP: coordination protocols that permit limited, externally validated self-modification.<n>This paper presents an exploratory systems feasibility study of Self-Evolving Coordination Protocols.
  arXiv  Detail & Related papers  (2026-02-02T14:45:04Z)
• Autonomous Chain-of-Thought Distillation for Graph-Based Fraud Detection [73.9189065770752]
  Graph-based fraud detection on text-attributed graphs (TAGs) requires jointly modeling rich textual semantics and relational dependencies.<n>We propose FraudCoT, a unified framework that advances TAG-based fraud detection through autonomous, graph-aware chain-of-thought (CoT) reasoning and scalable LLM-GNN co-training.
  arXiv  Detail & Related papers  (2026-01-30T13:12:12Z)
• Preventing the Collapse of Peer Review Requires Verification-First AI [49.995126139461085]
  We propose truth-coupling, i.e. how tightly venue scores track latent scientific truth.<n>We formalize two forces that drive a phase transition toward proxy-sovereign evaluation.
  arXiv  Detail & Related papers  (2026-01-23T17:17:32Z)
• Institutional AI: A Governance Framework for Distributional AGI Safety [1.3763052684269788]
  We identify three structural problems that emerge from core properties of AI models.<n>The solution is Institutional AI, a system-level approach that treats alignment as a question of effective governance of AI agent collectives.
  arXiv  Detail & Related papers  (2026-01-15T17:08:26Z)
• Towards a Science of Scaling Agent Systems [79.64446272302287]
  We formalize a definition for agent evaluation and characterize scaling laws as the interplay between agent quantity, coordination structure, modelic, and task properties.<n>We derive a predictive model using coordination metrics, that cross-validated R2=0, enabling prediction on unseen task domains.<n>We identify three effects: (1) a tool-coordination trade-off: under fixed computational budgets, tool-heavy tasks suffer disproportionately from multi-agent overhead, and (2) a capability saturation: coordination yields diminishing or negative returns once single-agent baselines exceed 45%.
  arXiv  Detail & Related papers  (2025-12-09T06:52:21Z)
• Making LLMs Reliable When It Matters Most: A Five-Layer Architecture for High-Stakes Decisions [51.56484100374058]
  Current large language models (LLMs) excel in verifiable domains where outputs can be checked before action but prove less reliable for high-stakes strategic decisions with uncertain outcomes.<n>This gap, driven by mutually cognitive biases in both humans and artificial intelligence (AI) systems, threatens the defensibility of valuations and sustainability of investments in the sector.<n>This report describes a framework emerging from systematic qualitative assessment across 7 frontier-grade LLMs and 3 market-facing venture vignettes under time pressure.
  arXiv  Detail & Related papers  (2025-11-10T22:24:21Z)
• Democracy-in-Silico: Institutional Design as Alignment in AI-Governed Polities [2.1485350418225244]
  Democracy-in-Silico is an agent-based simulation where societies of advanced AI agents govern themselves under different institutional frameworks.<n>We explore what it means to be human in an age of AI by tasking Large Language Models (LLMs) to embody agents with traumatic memories, hidden agendas, and psychological triggers.<n>We present a novel metric, the Power-Preservation Index (PPI), to quantify misaligned behavior where agents prioritize their own power over public welfare.
  arXiv  Detail & Related papers  (2025-08-27T04:44:41Z)
• Governance-as-a-Service: A Multi-Agent Framework for AI System Compliance and Policy Enforcement [0.0]
  We introduce Governance-as-a-Service (G): a policy-driven enforcement layer that regulates agent outputs at runtime.<n>G employs declarative rules and a Trust Factor mechanism that scores agents based on compliance and severity of violations.<n>Results show that G reliably blocks or redirects high-risk behaviors while preserving throughput.
  arXiv  Detail & Related papers  (2025-08-26T07:48:55Z)
• Watermarking Without Standards Is Not AI Governance [46.71493672772134]
  We argue that current implementations risk serving as symbolic compliance rather than delivering effective oversight.<n>We propose a three-layer framework encompassing technical standards, audit infrastructure, and enforcement mechanisms.
  arXiv  Detail & Related papers  (2025-05-27T18:10:04Z)
• Media and responsible AI governance: a game-theoretic and LLM analysis [61.132523071109354]
  This paper investigates the interplay between AI developers, regulators, users, and the media in fostering trustworthy AI systems.<n>Using evolutionary game theory and large language models (LLMs), we model the strategic interactions among these actors under different regulatory regimes.
  arXiv  Detail & Related papers  (2025-03-12T21:39:38Z)
• AGI, Governments, and Free Societies [0.0]
  We argue that AGI poses distinct risks of pushing societies toward either a 'despotic Leviathan' or an 'absent Leviathan'<n>We analyze how these dynamics could unfold through three key channels.<n> Enhanced state capacity through AGI could enable unprecedented surveillance and control, potentially entrenching authoritarian practices.<n>Conversely, rapid diffusion of AGI capabilities to non-state actors could undermine state legitimacy and governability.
  arXiv  Detail & Related papers  (2025-02-14T03:55:38Z)
• Unsupervised Full Constituency Parsing with Neighboring Distribution Divergence [48.69930912510414]
  We propose an unsupervised and training-free labeling procedure by exploiting the property of a recently introduced metric. For implementation, we develop NDD into Dual POS-NDD and build "molds" to detect constituents and their labels in sentences. We show that DP-NDD not only labels constituents precisely but also inducts more accurate unlabeled constituency trees than all previous unsupervised methods with simpler rules.
  arXiv  Detail & Related papers  (2021-10-29T17:27:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.