AI Alignment via Incentives and Correction

• URL: http://arxiv.org/abs/2605.01643v3
• Date: Mon, 11 May 2026 16:34:28 GMT
• Title: AI Alignment via Incentives and Correction
• Abstract summary: We study AI alignment through the lens of law-and-economics models of deterrence and enforcement.<n>A solver may benefit from producing a persuasive but incorrect answer, hiding uncertainty, or exploiting spurious shortcuts.<n>We formalize this interaction in a two-agent model in which a principal chooses rewards over joint correction outcomes.
• Score: 14.986111703734222
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We study AI alignment through the lens of law-and-economics models of deterrence and enforcement. In these models, misconduct is not treated as an external failure, but as a strategic response to incentives: an actor weighs the gain from violation against the probability of detection and the severity of punishment. We argue that the same logic arises naturally in agentic AI pipelines. A solver may benefit from producing a persuasive but incorrect answer, hiding uncertainty, or exploiting spurious shortcuts, while an auditor or verifier must decide whether costly monitoring is worthwhile. Alignment is therefore a fixed-point problem: stronger penalties may deter solver misbehavior, but they can also reduce the auditor's incentive to inspect, since auditing then mainly incurs cost on a population that appears increasingly aligned. This perspective also changes what should count as a post-training signal. Standard feedback often attaches reward to the final answer alone, but a solver-auditor pipeline exposes the full correction event: whether the solver erred, whether the auditor inspected, whether the error was caught, and whether oversight incentives remained active. We formalize this interaction in a two-agent model in which a principal chooses rewards over joint correction outcomes, inducing both solver behavior and auditor monitoring. Reward design is therefore a bilevel optimization problem: rewards are judged not by their immediate semantic meaning, but by the behavioral equilibrium they induce. We propose a bandit-based outer-loop procedure for searching over reward profiles using noisy interaction feedback. Experiments on an LLM coding pipeline show that adaptive reward profiles can maintain useful oversight pressure and improve principal-aligned outcomes relative to static hand-designed rewards, including a substantial reduction in hallucinated incorrect attempts.

Related papers

• RecurSE: Bounded Recursive Self-Evaluation for LLM Rubric Judges [48.831446371402414]
  RecurSE pairs a trainable judge with a synchronized policy-copy checker that audits the judge's reasoning against meta-rubrics to supply a scalar process reward.<n>We show that RecurSE achieves consistent generalization gains across held-out medical, pairwise, summarization, and professional benchmarks.
  arXiv  Detail & Related papers  (2026-08-25T08:35:04Z)
• Knowing When to Quit: Diagnosing and Training LLMs to Abort Futile Reasoning [109.33717747526583]
  Large language models generate computationally expensive yet semantically void reasoning on beyond-capability tasks.<n>We introduce textbfCaRL (textbfCapability-textbfaligned textbfReinforcement textbfL) to align behavior with capability boundaries.<n>Experiments demonstrate a substantial reduction in futile reasoning while preserving performance across task difficulties.
  arXiv  Detail & Related papers  (2026-07-31T09:30:33Z)
• Scaling, Lock-In, and Proxy Compliance: A Political Economy of Responsible AI [1.253312107729806]
  We develop a sequential political-economy model to address accountability for AI systems.<n>We show that an AI vendor chooses auditability and substantive mitigation, a deployer monitors after adoption while switching costs, and enforcement depends on verifiable evidence.<n>The results explain why documentation and standardized evaluations can coexist with persistent post-deployment harms.
  arXiv  Detail & Related papers  (2026-07-30T11:12:13Z)
• Signed Compression Progress on a Sealed Audit is Goodhart-Resistant [0.43830114853179497]
  Compression progress is a long-standing proposal for intrinsic motivation.<n>We prove that intrinsic reward is "credible" because it is paid only for learning.<n>We give a Lean 4 mechanization of the structural core.
  arXiv  Detail & Related papers  (2026-06-09T20:10:40Z)
• IMPACT-CYCLE: A Contract-Based Multi-Agent System for Claim-Level Supervisory Correction of Long-Video Semantic Memory [73.22944697933603]
  Existing pipelines produce opaque, end-to-end outputs that expose no intermediate state for inspection.<n>We present IMPACT-Cycle, a supervisory multi-agent system that reformulates long-video understanding as a multimodal iterative claim-level maintenance.
  arXiv  Detail & Related papers  (2026-04-22T03:03:33Z)
• Taming Actor-Observer Asymmetry in Agents via Dialectical Alignment [59.536125286960186]
  Multi-agent frameworks assigning specialized roles are increasingly adopted to enable self-reflection and mutual auditing.<n>We find it simultaneously induces a human-like cognitive bias known as Actor-Observer Asymmetry.<n>ReTAS guides agents to synthesize conflicting viewpoints into an objective consensus.
  arXiv  Detail & Related papers  (2026-04-21T15:05:58Z)
• Hodoscope: Unsupervised Monitoring for AI Misbehaviors [25.119538476586737]
  Existing approaches to monitoring AI agents rely on human-written rules or LLM-based judges that check for known failure modes.<n>We formulate unsupervised monitoring, drawing an analogy to unsupervised learning.<n>We introduce Hodoscope, a tool that operationalizes this insight.
  arXiv  Detail & Related papers  (2026-04-13T06:52:05Z)
• PRISM: Festina Lente Proactivity -- Risk-Sensitive, Uncertainty-Aware Deliberation for Proactive Agents [11.440330367799513]
  PRISM is a novel framework that couples a decision-theoretic gate with a dual-process reasoning architecture.<n>Inspired by festina lente (Latin: "make haste slowly"), we gate by an acceptance-calibrated, cost-derived threshold.<n>On ProactiveBench, PRISM reduces false alarms by 22.78% and improves F1 by 20.14% over strong baselines.
  arXiv  Detail & Related papers  (2026-02-02T01:56:29Z)
• Preventing the Collapse of Peer Review Requires Verification-First AI [49.995126139461085]
  We propose truth-coupling, i.e. how tightly venue scores track latent scientific truth.<n>We formalize two forces that drive a phase transition toward proxy-sovereign evaluation.
  arXiv  Detail & Related papers  (2026-01-23T17:17:32Z)
• Towards Robust Process Reward Modeling via Noise-aware Learning [33.1289107681179]
  We propose a two-stage framework to mitigate noisy supervision.<n>In the labeling stage, we introduce a reflection-aware label correction mechanism that uses a large language model (LLM) as a judge.<n>In the training stage, we propose a underlinetextbfIterative underlinetextbfTraining framework that enables the PRM to progressively refine noisy labels.
  arXiv  Detail & Related papers  (2026-01-19T06:03:58Z)
• Reinforcement Learning with Verifiable yet Noisy Rewards under Imperfect Verifiers [90.50039419576807]
  Reinforcement Learning with Verifiable Rewards (RLVR) trains policies against automated verifiers to avoid costly human labeling.<n>To reduce vulnerability to verifier hacking, many RLVR systems collapse rewards to binary $0,1$ during training.<n>This choice carries a cost: it introduces textitfalse negatives (rejecting correct answers, FNs) and textitfalse positives (accepting incorrect ones, FPs)
  arXiv  Detail & Related papers  (2025-10-01T13:56:44Z)
• Operator: A Protocol for Trustless Verification Under Uncertainty [0.0]
  We propose a protocol that enforces correctness through collateralized claims in a verification game.<n>Tasks are published as intents, and solvers compete to fulfill them.<n>Any challenger can challenge a result by staking against it to trigger the verification process.<n>Incorrect agents are slashed and correct opposition is rewarded, with an escalation path that penalizes erroneous verifiers themselves.
  arXiv  Detail & Related papers  (2025-07-01T10:22:35Z)
• Maximizing Information Gain in Partially Observable Environments via Prediction Reward [64.24528565312463]
  This paper tackles the challenge of using belief-based rewards for a deep RL agent. We derive the exact error between negative entropy and the expected prediction reward. This insight provides theoretical motivation for several fields using prediction rewards.
  arXiv  Detail & Related papers  (2020-05-11T08:13:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.