Calibrating Conservatism for Scalable Oversight

• URL: http://arxiv.org/abs/2605.28807v1
• Date: Wed, 27 May 2026 17:56:47 GMT
• Title: Calibrating Conservatism for Scalable Oversight
• Abstract summary: We introduce Calibrated Collective Oversight (CCO), which aggregates diverse auxiliary scoring functions into a penalty measuring deviation from a conservative baseline.<n>CCO calibrates this conservatism online using Conformal Decision Theory, ensuring that undesirable outcomes remain below a user-specified target threshold.<n>On a modified version of SWE-bench, weaker overseers successfully constrain an adversarially misaligned stronger agent; on MACHIAVELLI, CCO substantially reduces ethical violations while preserving reward.
• Score: 9.553819152637493
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Agentic AI systems capable of autonomous planning and extended environmental interaction pose a fundamental control problem: how can humans maintain meaningful oversight of systems that may exceed their own capabilities? Existing approaches to scalable oversight rely on complex assumptions, remain largely heuristic, or lack practical methods for sequential settings with statistical guarantees. We introduce Calibrated Collective Oversight (CCO), which aggregates diverse auxiliary scoring functions into a penalty measuring deviation from a conservative baseline. Inspired by Attainable Utility Preservation, CCO enables collective conservatism: actions face a penalty proportional to overseer concern, so high-utility actions are still selected when overseers find them unobjectionable and overridden only when concern accumulates. CCO calibrates this conservatism online using Conformal Decision Theory, ensuring that undesirable outcomes remain below a user-specified target threshold with finite-time bounds and no distributional assumptions. On a modified version of SWE-bench, weaker overseers successfully constrain an adversarially misaligned stronger agent; on MACHIAVELLI, CCO substantially reduces ethical violations while preserving reward. In both settings, empirical violation rates closely match the specified targets, as predicted by the theory.

Related papers

• Conformal Changepoint Localization and Root Cause Analysis with Corrupted Observations [55.76952683833966]
  This paper proposes weighted CONCH (W-CONCH) and weighted CROC (W-CROC) to reduce confidence set size when data may be corrupted.<n>Experiments on image-based and real-world changepoint and root-cause benchmarks show that uncertainty-based weighting substantially reduces confidence set size.
  arXiv  Detail & Related papers  (2026-07-29T05:16:59Z)
• Integrity-Gated Eco-CACC: Epistemic Admissibility for Cooperative Driving at Signalized Intersections [0.3867363075280543]
  This letter proposes an Integrity-Gated Eco-CACC framework that explicitly monitors the consistency between internal vehicle beliefs and external sensing.<n>The resulting trust score regulates control authority, enabling a transition between nominal eco-driving and a safety-dominant fallback maneuver.
  arXiv  Detail & Related papers  (2026-07-20T22:59:52Z)
• Predictive Objectives Discard Exogenous Control-Relevant Features: A Controlled Mechanistic Study [0.0]
  Joint-embedding predictive (JEPA) objectives learn representations by predicting future latents.<n>This occurs because the objective optimize temporal predictability rather than control-relevance.<n>We show that as little as 2% of reward-labeled transitions recovers the feature, the effect holds across two environments.
  arXiv  Detail & Related papers  (2026-06-29T10:00:50Z)
• Does Reasoning Preserve Alignment? On the Trustworthiness of Large Reasoning Models [55.788110316999166]
  We compare reasoning models produced via supervised fine-tuning, RL-based post-training, and distillation against instruction-tuned baselines.<n>We observe that reasoning models often improve on reasoning benchmarks but exhibit alignment regressions.<n>These regressions are consistent with behavioral drift from the instruction-tuned baseline, measured by KL divergence.
  arXiv  Detail & Related papers  (2026-06-09T16:14:27Z)
• When Outcome Looks Right But Discipline Fails: Trace-Based Evaluation Under Hidden Competitor State [0.39287497907611874]
  Outcome-only evaluation can certify economically unsafe agents.<n>In hotel pricing with hidden competitor state, a learner can achieve plausible revenue per available room while failing to preserve the rate discipline of a rule-based revenue-management policy.<n>We introduce discipline stability, a trace-based evaluation paradigm.
  arXiv  Detail & Related papers  (2026-05-18T15:58:34Z)
• Semantic Reward Collapse and the Preservation of Epistemic Integrity in Adaptive AI Systems [0.0]
  Recent advances in reinforcement learning have substantially improved the usability, coherence, and safety of large language models.<n> recurring behaviors such as performative certainty suggest unresolved structural issues within scalarized preference optimization systems.<n>We propose Semantic Reward Collapse (SRC), the compression of semantically distinct forms of evaluative dissatisfaction into generalized optimization signals.
  arXiv  Detail & Related papers  (2026-05-12T17:03:26Z)
• Mechanical Conscience: A Mathematical Framework for Dependability of Machine Intelligenc [0.0]
  This paper introduces mechanical conscience (MC), a novel concept and simplified mathematical framework.<n>MC operationalizes trajectory-level normative regulation for both single-agent and distributed intelligent systems.<n>Core theoretical properties are established: admissibility equivalence, existence of optimal regulation, and monotonic deviation reduction.
  arXiv  Detail & Related papers  (2026-05-05T15:14:02Z)
• Learning from Emptiness: De-biasing Listwise Rerankers with Content-Agnostic Probability Calibration [76.08899010904652]
  CapCal is a training-free framework that mechanically decouples positional bias from ranking decisions.<n>It achieves superior performance among training-free methods while preserving single-pass efficiency.
  arXiv  Detail & Related papers  (2026-04-11T10:47:22Z)
• CMP: Robust Whole-Body Tracking for Loco-Manipulation via Competence Manifold Projection [64.65196237023754]
  Manifold Competence Projection improves robustness against Out-of-Distribution perturbations.<n>We employ a Frame-Wise Safety Scheme that transforms the infinite-horizon safety constraint into a computationally efficient single-step manifold inclusion.<n>Experiments demonstrate that CMP achieves up to a 10-fold survival rate improvement in typical OOD scenarios.
  arXiv  Detail & Related papers  (2026-04-08T18:00:39Z)
• Authority-Level Priors: An Under-Specified Constraint in Hierarchical Predictive Processing [0.0]
  Authority-Level Priors (ALPs) are meta-structural constraints defining a regulatory-admissible subset of identity-level hypotheses.<n>ALPs are not additional representational states nor hyperpriors over precision; they constrain which hypotheses are admissible for regulatory control.<n>A computational formalisation restricts policy optimisation to policies generated by authorised hypotheses.
  arXiv  Detail & Related papers  (2026-03-19T13:27:47Z)
• Core Safety Values for Provably Corrigible Agents [2.6451153531057985]
  We introduce the first implementable framework for corrigibility, with provable guarantees in multi-step, partially observed environments.<n>Our framework replaces a single reward with five *structurally separate* utility heads.<n>For open-ended settings where adversaries can modify the agent, we prove that deciding whether an arbitrary post-hack agent will ever violate corrigibility is undecidable.
  arXiv  Detail & Related papers  (2025-07-28T16:19:25Z)
• COIN: Uncertainty-Guarding Selective Question Answering for Foundation Models with Provable Risk Guarantees [51.5976496056012]
  COIN is an uncertainty-guarding selection framework that calibrates statistically valid thresholds to filter a single generated answer per question.<n>COIN estimates the empirical error rate on a calibration set and applies confidence interval methods to establish a high-probability upper bound on the true error rate.<n>We demonstrate COIN's robustness in risk control, strong test-time power in retaining admissible answers, and predictive efficiency under limited calibration data.
  arXiv  Detail & Related papers  (2025-06-25T07:04:49Z)
• Criticality and Safety Margins for Reinforcement Learning [53.10194953873209]
  We seek to define a criticality framework with both a quantifiable ground truth and a clear significance to users.<n>We introduce true criticality as the expected drop in reward when an agent deviates from its policy for n consecutive random actions.<n>We also introduce the concept of proxy criticality, a low-overhead metric that has a statistically monotonic relationship to true criticality.
  arXiv  Detail & Related papers  (2024-09-26T21:00:45Z)
• Exterior Penalty Policy Optimization with Penalty Metric Network under Constraints [52.37099916582462]
  In Constrained Reinforcement Learning (CRL), agents explore the environment to learn the optimal policy while satisfying constraints. We propose a theoretically guaranteed penalty function method, Exterior Penalty Policy Optimization (EPO), with adaptive penalties generated by a Penalty Metric Network (PMN) PMN responds appropriately to varying degrees of constraint violations, enabling efficient constraint satisfaction and safe exploration.
  arXiv  Detail & Related papers  (2024-07-22T10:57:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.