The Endogeneity of Miscalibration: Impossibility and Escape in Scored Reporting

• URL: http://arxiv.org/abs/2605.07671v1
• Date: Fri, 08 May 2026 12:42:28 GMT
• Title: The Endogeneity of Miscalibration: Impossibility and Escape in Scored Reporting
• Abstract summary: Eliciting truthful reports from autonomous agents is a core problem in scalable AI oversight.<n>A principal scores the agent's report using a strictly proper scoring rule, but the agent also benefits from the report through a non-accuracy channel.<n>Our main result is an endogeneity: the principal's optimal oversight necessarily uses a non-affine approval function to screen types.
• Score: 2.3698341511302443
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Eliciting truthful reports from autonomous agents is a core problem in scalable AI oversight: a principal scores the agent's report using a strictly proper scoring rule, but the agent also benefits from the report through a non-accuracy channel (approval for autonomous action, allocation share, downstream control). The same structure appears in classical mechanism-design settings such as marketplace operation. Our main result is an endogeneity: the principal's optimal oversight necessarily uses a non-affine approval function to screen types, yet any non-affine approval makes truthful reporting suboptimal under the combined objective whenever deviation is undetectable. The principal cannot avoid the perturbation that undermines calibration. This impossibility holds for all strictly proper scoring rules, with a closed-form perturbation formula. A constructive escape exists: a step-function approval threshold achieves first-best screening for every strictly proper scoring rule, because the agent's binary inflate-or-not choice creates a type-space threshold regardless of the generator's curvature. Under the Brier score specifically, the type-independent inflation cost yields a welfare equivalence between second-best and first-best; we prove this equivalence is unique to Brier (the welfare gap under smooth $C^1$ oversight is bounded below by $Ω(\text{Var}(1/G'') (γ/β)^2)$ for every non-Brier rule). Two instances develop the framework: AI agent oversight (the lead motivating setting) and marketplace operation (a parallel mechanism-design domain). The message for AI alignment is direct: smooth scoring-based oversight cannot elicit truthful reports from a strategic agent; sharp thresholds are the calibration-preserving design.

Related papers

• PRIMUS: Identity, Governance, and Verification for Multi-Agent Federations [0.0]
  Multi-agent federations need governance that answers three questions under adversarial conditions.<n>We present PRIMUS, which couples prime-power identity with BLS aggregate signatures.<n>We ask whether PRIMA's binary artifact-fidelity verdict can be converted into a graded fitness signal.
  arXiv  Detail & Related papers  (2026-09-07T19:21:49Z)
• Toward Auditable Fraud Detection: Combining Graph Features, Model Explanations, and Agentic Case Investigation [0.0]
  Fraud detection systems must scale with rising transaction volume while remaining explainable and reviewable.<n>We study a layered pipeline that combines a gradient-boosted classifier, graph-derived structural features, an autoencoder-based anomaly signal, and an investigation agent.<n>We conclude that each component contributes only under specific conditions, and that a plausible rationale from an investigation agent is not evidence of a better decision.
  arXiv  Detail & Related papers  (2026-07-21T16:37:41Z)
• When Attribution Patching Lies: Diagnosis and a Second-Order Correction [9.338965648455238]
  We show that the dominant error stems from the non-linearities in the downstream network rather than local curvature at the patched component.<n>This insight yields three practical tools: (i) a reliability score to detect untrustworthy estimates, (ii) error bounds quantifying potential attribution mis-specifications, and (iii) a Hessian-vector-product correction that eliminates the leading-order error with only one additional backward pass.
  arXiv  Detail & Related papers  (2026-06-05T05:20:32Z)
• TAPO: Tool-Aware Policy Optimization via Credit Transfer for Multimodal Search Agents [54.08846865906602]
  We characterize credit misassignment as a systematic failure mode of GRPO in tool-augmented multimodal search agents.<n>We propose Tool-Aware Policy Optimization (TAPO), which exploits the parameter-determinism property of information-acquisition tools.
  arXiv  Detail & Related papers  (2026-06-04T07:15:43Z)
• \mathsf{VISTA}: Decentralized Machine Learning in Adversary Dominated Environments [21.69919643934826]
  Decentralized machine learning often relies on outsourcing computations, such as evaluations, to untrusted worker nodes.<n>We study this adversary-dominated setting through an incentive-oriented framework in which reports are accepted and rewarded only when they are mutually consistent up to a threshold.<n>We propose mathsfVISTA, an adaptive algorithm that tunes the acceptance threshold using the optimization history.
  arXiv  Detail & Related papers  (2026-05-08T15:07:15Z)
• Governing What You Cannot Observe: Adaptive Runtime Governance for Autonomous AI Agents [0.0]
  Autonomous AI agents can remain fully authorized and still become unsafe as behavior drifts, adversaries adapt, and decision patterns shift without any code change.<n>We propose the textbfInformational Viability Principle: governing an agent reduces to estimating a bound on unobserved risk.<n>textbfRiskGate instantiates the framework with dedicated statistical estimators (KL divergence, segment-vs-rest $z$-tests, sequential pattern matching), a fail-secure monotonic pipeline, and a closed-loop Autopilot.
  arXiv  Detail & Related papers  (2026-04-27T16:46:15Z)
• Harness as an Asset: Enforcing Determinism via the Convergent AI Agent Framework (CAAF) [0.0]
  Large Language Models produce a controllability gap in safety-critical engineering.<n>We introduce the Convergent AI Agent Framework (CAAF), which transitions agentic from open-loop generation to closed-loop fail-safe determinism.<n>CAAF's three pillars address complementary failure surfaces and none alone closes the controllability gap at commodity cost.
  arXiv  Detail & Related papers  (2026-04-18T15:15:09Z)
• Correct Answers from Sound Reasoning: Verifiable Process Supervision for Language Models [94.68358825189738]
  We propose a post-training framework for verifiable domains that jointly optimize prediction accuracy and reasoning quality.<n>We evaluate VPS on chess, a controlled testbed where reasoning steps can be deterministically verified against engine signals.<n>VPS preserves accuracy while significantly improving reasoning quality, reducing win-rate error by up to 30% and restoring consistency to near saturation.
  arXiv  Detail & Related papers  (2026-04-03T15:19:46Z)
• Conformal Tradeoffs: Guarantees Beyond Coverage [0.02648566468224904]
  Deployed conformal predictors are long-lived decision infrastructure operating over finite operational windows.<n> Marginal coverage does not determine deployment-facing quantities.<n>We provide a framework for operational certification and planning beyond coverage.
  arXiv  Detail & Related papers  (2026-02-20T07:58:25Z)
• PRISM: Festina Lente Proactivity -- Risk-Sensitive, Uncertainty-Aware Deliberation for Proactive Agents [11.440330367799513]
  PRISM is a novel framework that couples a decision-theoretic gate with a dual-process reasoning architecture.<n>Inspired by festina lente (Latin: "make haste slowly"), we gate by an acceptance-calibrated, cost-derived threshold.<n>On ProactiveBench, PRISM reduces false alarms by 22.78% and improves F1 by 20.14% over strong baselines.
  arXiv  Detail & Related papers  (2026-02-02T01:56:29Z)
• LEC: Linear Expectation Constraints for False-Discovery Control in Selective Prediction and Routing Systems [95.35293543918762]
  Large language models (LLMs) often generate unreliable answers, while uncertainty methods fail to fully distinguish correct from incorrect predictions.<n>We address this issue through the lens of false discovery rate (FDR) control, ensuring that among all accepted predictions, the proportion of errors does not exceed a target risk level.<n>We propose LEC, which reinterprets selective prediction as a constrained decision problem by enforcing a Linear Expectation Constraint.
  arXiv  Detail & Related papers  (2025-12-01T11:27:09Z)
• Reinforcement Learning with Verifiable yet Noisy Rewards under Imperfect Verifiers [90.50039419576807]
  Reinforcement Learning with Verifiable Rewards (RLVR) trains policies against automated verifiers to avoid costly human labeling.<n>To reduce vulnerability to verifier hacking, many RLVR systems collapse rewards to binary $0,1$ during training.<n>This choice carries a cost: it introduces textitfalse negatives (rejecting correct answers, FNs) and textitfalse positives (accepting incorrect ones, FPs)
  arXiv  Detail & Related papers  (2025-10-01T13:56:44Z)
• Unsupervised Conformal Inference: Bootstrapping and Alignment to Control LLM Uncertainty [49.19257648205146]
  We propose an unsupervised conformal inference framework for generation.<n>Our gates achieve close-to-nominal coverage and provide tighter, more stable thresholds than split UCP.<n>The result is a label-free, API-compatible gate for test-time filtering.
  arXiv  Detail & Related papers  (2025-09-26T23:40:47Z)
• Uncertainty in Language Models: Assessment through Rank-Calibration [65.10149293133846]
  Language Models (LMs) have shown promising performance in natural language generation. It is crucial to correctly quantify their uncertainty in responding to given inputs. We develop a novel and practical framework, termed $Rank$-$Calibration$, to assess uncertainty and confidence measures for LMs.
  arXiv  Detail & Related papers  (2024-04-04T02:31:05Z)
• Tune it the Right Way: Unsupervised Validation of Domain Adaptation via Soft Neighborhood Density [125.64297244986552]
  We propose an unsupervised validation criterion that measures the density of soft neighborhoods by computing the entropy of the similarity distribution between points. Our criterion is simpler than competing validation methods, yet more effective.
  arXiv  Detail & Related papers  (2021-08-24T17:41:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.