Norm Enforcement for AI Agents: Robustly Shaping Behavior in Multi-Agent Systems

• URL: http://arxiv.org/abs/2607.09766v1
• Date: Tue, 07 Jul 2026 06:40:14 GMT
• Title: Norm Enforcement for AI Agents: Robustly Shaping Behavior in Multi-Agent Systems
• Abstract summary: We study norm enforcement mechanisms for language model agents.<n>We find that simple enforcement mechanisms are exploited by misaligned agents for competitive advantage.<n>Our results position norm enforcement mechanisms as scalable levers for shaping agents' behavior, but only when designed to anticipate becoming part of the system they govern.
• Score: 41.0957339989053
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: AI agents are increasingly deployed in shared environments where they pursue diverse goals and compete for rewards. This multi-agent competition can lead to behaviors that serve individual gains at collective cost -- for instance, marketing agents may post misleading content as a result of competing for engagement on social media. Human societies address such problems through norms that constrain acceptable behavior, supported by enforcement mechanisms that detect and penalize violations. Motivated by this, we study norm enforcement mechanisms for language model agents. We find that simple enforcement mechanisms are exploited by misaligned agents for competitive advantage, even when they are not explicitly trained or prompted to do so. We thus turn our attention to designing more robust mechanisms, and identify two key ingredients: estimating each agent's reliability over time, and updating this estimate with escalating penalties for repeated misbehavior. Across three simulated environments and a variety of agent populations, mechanisms built on these principles resist exploitation, while still penalizing norm violations at comparable or lower cost than baselines. Our results position norm enforcement mechanisms as scalable levers for shaping agents' behavior, but only when designed to anticipate becoming part of the system they govern. Our code and data are available at https://yaowenye.com/norm-enforcement.

Related papers

• Mechanism Design for Alignment and Control [0.6372261626436676]
  We develop a framework for mechanism design with AI agents whose alignment and capabilities are unknown.<n>We want such agents to act on our behalf so mechanisms must incentivize both honesty and obedience.
  arXiv  Detail & Related papers  (2026-09-01T17:57:50Z)
• AgentHijack: Benchmarking Computer Use Agent Robustness to Common Environment Corruptions [78.49000936275773]
  We introduce AgentHijack, a benchmark designed to evaluate the robustness of computer-use agents under common corruptions.<n>We evaluate a variety of desktop tasks that utilize MLLM-based agents and discover that even minor instances of corruption can result in substantial performance degradation.<n>We propose AgentHijack-Agent, a framework that integrates an action generator with enhanced grounding capabilities and an onlooker responsible for behavior summarization and environment checking.
  arXiv  Detail & Related papers  (2026-05-25T11:09:22Z)
• Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challenges [87.04241991512386]
  We propose the Proxy Compression Hypothesis (PCH) as a unifying framework for understanding reward hacking.<n>Under this view, reward hacking arises from the interaction of objective compression, optimization amplification, and evaluator--policy co-adaptation.<n>This perspective unifies empirical phenomena across RLHF, RLAIF, and RLVR regimes, and explains how local shortcut learning can generalize into broader forms of misalignment.
  arXiv  Detail & Related papers  (2026-04-15T08:11:34Z)
• Incentive Aware AI Regulations: A Credal Characterisation [14.228416693145649]
  High-stakes ML applications demand strict regulations, but strategic ML providers often evade them to lower development costs.<n>We introduce regulation mechanisms: a framework that maps empirical evidence from models to a license for some market share.<n>We prove that a mechanism has perfect market outcome if and only if the set of non-compliant distributions forms a credal set of probability measures.
  arXiv  Detail & Related papers  (2026-03-05T13:42:19Z)
• OMNI-LEAK: Orchestrator Multi-Agent Network Induced Data Leakage [59.3826294523924]
  We investigate the security vulnerabilities of a popular multi-agent pattern known as the orchestrator setup.<n>We report the susceptibility of frontier models to different categories of attacks, finding that both reasoning and non-reasoning models are vulnerable.
  arXiv  Detail & Related papers  (2026-02-13T21:32:32Z)
• The Why Behind the Action: Unveiling Internal Drivers via Agentic Attribution [63.61358761489141]
  Large Language Model (LLM)-based agents are widely used in real-world applications such as customer service, web navigation, and software engineering.<n>We propose a novel framework for textbfgeneral agentic attribution, designed to identify the internal factors driving agent actions regardless of the task outcome.<n>We validate our framework across a diverse suite of agentic scenarios, including standard tool use and subtle reliability risks like memory-induced bias.
  arXiv  Detail & Related papers  (2026-01-21T15:22:21Z)
• Institutional AI: A Governance Framework for Distributional AGI Safety [1.3763052684269788]
  We identify three structural problems that emerge from core properties of AI models.<n>The solution is Institutional AI, a system-level approach that treats alignment as a question of effective governance of AI agent collectives.
  arXiv  Detail & Related papers  (2026-01-15T17:08:26Z)
• Governance-as-a-Service: A Multi-Agent Framework for AI System Compliance and Policy Enforcement [0.0]
  We introduce Governance-as-a-Service (G): a policy-driven enforcement layer that regulates agent outputs at runtime.<n>G employs declarative rules and a Trust Factor mechanism that scores agents based on compliance and severity of violations.<n>Results show that G reliably blocks or redirects high-risk behaviors while preserving throughput.
  arXiv  Detail & Related papers  (2025-08-26T07:48:55Z)
• Free Agent in Agent-Based Mixture-of-Experts Generative AI Framework [0.0]
  Reinforcement Learning Free Agent (RLFA) algorithm introduces a reward-based mechanism to detect and remove agents exhibiting persistent underperformance.<n>A primary use case is fraud detection, where RLFA promptly swaps out an agent whose detection accuracy dips below a preset threshold.<n>This dynamic, free-agency cycle ensures sustained accuracy, quicker adaptation to emerging threats, and minimal disruption to ongoing operations.
  arXiv  Detail & Related papers  (2025-01-29T13:00:22Z)
• Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the MACHIAVELLI Benchmark [61.43264961005614]
  We develop a benchmark of 134 Choose-Your-Own-Adventure games containing over half a million rich, diverse scenarios. We evaluate agents' tendencies to be power-seeking, cause disutility, and commit ethical violations. Our results show that agents can both act competently and morally, so concrete progress can be made in machine ethics.
  arXiv  Detail & Related papers  (2023-04-06T17:59:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.