Mechanism Design for Alignment and Control

• URL: http://arxiv.org/abs/2609.01595v1
• Date: Tue, 01 Sep 2026 17:57:50 GMT
• Title: Mechanism Design for Alignment and Control
• Abstract summary: We develop a framework for mechanism design with AI agents whose alignment and capabilities are unknown.<n>We want such agents to act on our behalf so mechanisms must incentivize both honesty and obedience.
• Score: 0.6372261626436676
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We develop a framework for mechanism design with AI agents whose alignment (preferences) and capabilities (feasible actions and information) are unknown. We want such agents to act on our behalf so mechanisms must incentivize both honesty and obedience. A one-sided imitation structure---capabilities can be concealed but not counterfeited---yields a revelation principle, a characterization of implementable policies via nested cyclical monotonicity, and conditions under which eliciting higher-order beliefs can discipline multiple agents. We apply our framework to stylized examples of (i) sandbagging in which a more capable agent pretends to be less capable; (ii) an alignment--interpretability trade-off, where the two are substitutes in the instrument but complements in value; (iii) discipline via peer scoring; (iv) coupling rewards to induce competition among multiple agents; and (v) scalable oversight and reward shaping.

Related papers

• Norm Enforcement for AI Agents: Robustly Shaping Behavior in Multi-Agent Systems [41.0957339989053]
  We study norm enforcement mechanisms for language model agents.<n>We find that simple enforcement mechanisms are exploited by misaligned agents for competitive advantage.<n>Our results position norm enforcement mechanisms as scalable levers for shaping agents' behavior, but only when designed to anticipate becoming part of the system they govern.
  arXiv  Detail & Related papers  (2026-07-07T06:40:14Z)
• Provably Secure Agent Guardrail [89.79561918065122]
  Existing defense architectures rely on empirical semantic guardrails and probabilistic large model adjudicators.<n>This paper proposes a new security paradigm for agents based on the fundamental limitations of logical reasoning.
  arXiv  Detail & Related papers  (2026-05-28T02:12:41Z)
• Taming Actor-Observer Asymmetry in Agents via Dialectical Alignment [59.536125286960186]
  Multi-agent frameworks assigning specialized roles are increasingly adopted to enable self-reflection and mutual auditing.<n>We find it simultaneously induces a human-like cognitive bias known as Actor-Observer Asymmetry.<n>ReTAS guides agents to synthesize conflicting viewpoints into an objective consensus.
  arXiv  Detail & Related papers  (2026-04-21T15:05:58Z)
• Beyond Arrow's Impossibility: Fairness as an Emergent Property of Multi-Agent Collaboration [3.4539478661465766]
  As large language models become agentic, we propose that fairness emerges through interaction and exchange.<n>We study this via a controlled hospital triage framework in which two agents negotiate over three structured debate rounds.
  arXiv  Detail & Related papers  (2026-04-15T10:34:35Z)
• Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challenges [87.04241991512386]
  We propose the Proxy Compression Hypothesis (PCH) as a unifying framework for understanding reward hacking.<n>Under this view, reward hacking arises from the interaction of objective compression, optimization amplification, and evaluator--policy co-adaptation.<n>This perspective unifies empirical phenomena across RLHF, RLAIF, and RLVR regimes, and explains how local shortcut learning can generalize into broader forms of misalignment.
  arXiv  Detail & Related papers  (2026-04-15T08:11:34Z)
• Incentive Aware AI Regulations: A Credal Characterisation [14.228416693145649]
  High-stakes ML applications demand strict regulations, but strategic ML providers often evade them to lower development costs.<n>We introduce regulation mechanisms: a framework that maps empirical evidence from models to a license for some market share.<n>We prove that a mechanism has perfect market outcome if and only if the set of non-compliant distributions forms a credal set of probability measures.
  arXiv  Detail & Related papers  (2026-03-05T13:42:19Z)
• Agentic Uncertainty Quantification [76.94013626702183]
  We propose a unified Dual-Process Agentic UQ (AUQ) framework that transforms verbalized uncertainty into active, bi-directional control signals.<n>Our architecture comprises two complementary mechanisms: System 1 (Uncertainty-Aware Memory, UAM), which implicitly propagates verbalized confidence and semantic explanations to prevent blind decision-making; and System 2 (Uncertainty-Aware Reflection, UAR), which utilizes these explanations as rational cues to trigger targeted inference-time resolution only when necessary.
  arXiv  Detail & Related papers  (2026-01-22T07:16:26Z)
• Institutional AI: A Governance Framework for Distributional AGI Safety [1.3763052684269788]
  We identify three structural problems that emerge from core properties of AI models.<n>The solution is Institutional AI, a system-level approach that treats alignment as a question of effective governance of AI agent collectives.
  arXiv  Detail & Related papers  (2026-01-15T17:08:26Z)
• From Competition to Coordination: Market Making as a Scalable Framework for Safe and Aligned Multi-Agent LLM Systems [5.165179548592513]
  We introduce a market-making framework for multi-agent large language model (LLM) coordination.<n>In this setup, each agent acts as a market participant, updating and trading probabilistic beliefs, to converge toward shared, truthful outcomes.<n> Empirically, we evaluate this approach across factual reasoning, ethical judgment, and commonsense inference tasks.
  arXiv  Detail & Related papers  (2025-11-18T16:47:15Z)
• SONA: Learning Conditional, Unconditional, and Mismatching-Aware Discriminator [54.562217603802075]
  We introduce Sum of Naturalness and Alignment (SONA), which employs separate projections for naturalness (authenticity) and alignment in the final layer with an inductive bias.<n>Experiments on class-conditional generation tasks show thatSONA achieves superior sample quality and conditional alignment compared to state-of-the-art methods.
  arXiv  Detail & Related papers  (2025-10-06T08:26:06Z)
• VCG Mechanism Design with Unknown Agent Values under Stochastic Bandit Feedback [104.06766271716774]
  We study a multi-round welfare-maximising mechanism design problem in instances where agents do not know their values. We first define three notions of regret for the welfare, the individual utilities of each agent and that of the mechanism. Our framework also provides flexibility to control the pricing scheme so as to trade-off between the agent and seller regrets.
  arXiv  Detail & Related papers  (2020-04-19T18:00:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.