Signed Compression Progress on a Sealed Audit is Goodhart-Resistant

• URL: http://arxiv.org/abs/2606.11417v1
• Date: Tue, 09 Jun 2026 20:10:40 GMT
• Title: Signed Compression Progress on a Sealed Audit is Goodhart-Resistant
• Abstract summary: Compression progress is a long-standing proposal for intrinsic motivation.<n>We prove that intrinsic reward is "credible" because it is paid only for learning.<n>We give a Lean 4 mechanization of the structural core.
• Score: 0.43830114853179497
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Compression progress is a long-standing proposal for intrinsic motivation: reward an agent when its world model becomes better at predicting or compressing experience. The folk claim is that this reward is "credible" because it is paid only for learning. We make this precise and prove it. If intrinsic reward is the signed decrease of a fixed sealed-audit loss, r_t = E(theta_{t-1}) - E(theta_t), then cumulative reward telescopes exactly to endpoint audit improvement, so no policy can push reward up indefinitely while true audit performance stagnates or degrades. For finite audit panels the same result holds with a sharp false-positive budget: cumulative empirical reward is at most true audit improvement plus 2 Delta_n(F, delta), the uniform audit deviation of the model class. This is horizon-free: adaptivity over time costs nothing once the sealed panel uniformly controls the class. The theorem also identifies the failure modes: the guarantee disappears if progress is clipped, scored on the agent's own stream, exposed to a high-capacity model on a reusable panel, or applied to a neural class that makes Delta_n vacuous. We give a Lean 4 mechanization of the structural core (telescoping, the finite-audit bound, finite Gibbs, and the entropy floor) and an experiment suite on ARC-TGI grid-transformation generators with adaptive holdout attacks. Experiments confirm the theory: finite-audit deviation scales as n^{-0.527}; signed progress resists clip-farming, stream leakage, and noisy-TV curiosity; naive reusable audits are exploitable by black-box scalar feedback, while standard release defenses keep the attack below the 2 Delta_n threshold. Signed compression progress on a sealed audit is an accounting signal of genuine improvement.

Related papers

• SynthGuard-ReleaseBench: Locked-Audit Evidence for Synthetic Tabular Data Releases [0.0]
  We introduce SynthGuard-ReleaseBench, an audit framework that locks the use, candidate panel, tolerances, and audit schedule before evaluation.<n>It compares real-trained and synthetic-trained on protected data, gives simultaneous finite-sample bounds for loss gaps, requires controls, and keeps utility, empirical privacy risk mechanism claims separate.
  arXiv  Detail & Related papers  (2026-08-14T03:12:17Z)
• Agents That Certify Their Own Exploits: Confidence-Scheduled Restricted Responses for Safe Opponent Exploitation [38.06569764716213]
  We introduce emphbudget-constrained confidence-scheduled restricted responses (CS-RNR)<n>CS-RNR is the first opponent-exploitation method whose safety guarantee is a certificate the agent computes on the strategy it actually deploys.
  arXiv  Detail & Related papers  (2026-07-30T16:57:57Z)
• Counterfactual Shapley Credit Assignment [49.39649767581182]
  Credit Assignment Problem (CAP) is fundamental to developing efficient and explainable Reinforcement Learning (RL) agents.<n>We introduce Counterfactual Shapley Credit Assignment, a novel framework that attributes credit and blame via the Counterfactual Shapley ($$-value)<n>We derive a consistent estimator that computes $$$-values, enabling a new class of policy methods, $$-PPO, combined with Prioritized Trajectory Replay (PTR)
  arXiv  Detail & Related papers  (2026-07-18T23:28:20Z)
• A Benchmark for Strategic Auditee Gaming Under Continuous Compliance Monitoring [1.253312107729806]
  Continuous post-deployment compliance audits create a class of strategic gaming.<n>We formalize continuous auditing as a $T$-round Stackelberg game between an auditor that commits to a temporal policy and an adaptive auditee.
  arXiv  Detail & Related papers  (2026-05-07T14:28:34Z)
• AI Alignment via Incentives and Correction [14.986111703734222]
  We study AI alignment through the lens of law-and-economics models of deterrence and enforcement.<n>A solver may benefit from producing a persuasive but incorrect answer, hiding uncertainty, or exploiting spurious shortcuts.<n>We formalize this interaction in a two-agent model in which a principal chooses rewards over joint correction outcomes.
  arXiv  Detail & Related papers  (2026-05-02T23:28:02Z)
• Unleashing Implicit Rewards: Prefix-Value Learning for Distribution-Level Optimization [74.91418266859297]
  Implicit process reward models (PRMs) provide fine-grained reward signals along the reasoning process.<n>PRMs suffer from a train-inference mismatch: training only constrains a sequence-level aggregate, whereas inference requires token-level scores to reflect local step quality.<n>We propose a novel Implicit Prefix-Value Reward Model (IPVRM), which directly learns a prefix-conditioned value function estimating the probability of eventual correctness.<n>We also propose Distribution-Level RL (DistRL), which computes TD advantages for both sampled tokens and high-probability candidate tokens.
  arXiv  Detail & Related papers  (2026-04-14T18:19:54Z)
• Reward Under Attack: Analyzing the Robustness and Hackability of Process Reward Models [68.45272703833209]
  We show that state-of-the-art PRMs are systematically exploitable under adversarial optimization pressure.<n>We introduce a three-tiered diagnostic framework that applies increasing adversarial pressure to quantify these vulnerabilities.<n>We release PRM-BiasBench and a diagnostic toolkit to enable robustness evaluation before deployment.
  arXiv  Detail & Related papers  (2026-02-20T23:38:03Z)
• Preventing the Collapse of Peer Review Requires Verification-First AI [49.995126139461085]
  We propose truth-coupling, i.e. how tightly venue scores track latent scientific truth.<n>We formalize two forces that drive a phase transition toward proxy-sovereign evaluation.
  arXiv  Detail & Related papers  (2026-01-23T17:17:32Z)
• CARE What Fails: Contrastive Anchored-REflection for Verifiable Multimodal [84.71254539482369]
  Group-relative reinforcement learning with verifiable rewards (RLVR) often wastes the most informative data it already has the failures.<n>We present CARE, a failure-centric post-training framework for multimodal reasoning that turns errors into supervision.<n> CARE improves accuracy and training smoothness while explicitly increasing the share of learning signal that comes from failures.
  arXiv  Detail & Related papers  (2025-12-22T16:34:21Z)
• Hybrid Reinforcement: When Reward Is Sparse, It's Better to Be Dense [36.71358559780692]
  HERO is a reinforcement learning framework that integrates verifier signals with reward-model scores in a structured way.<n> HERO consistently outperforms RM-only and verifier-only baselines, with strong gains on both verifiable and hard-to-verify tasks.
  arXiv  Detail & Related papers  (2025-10-08T17:09:41Z)
• Reinforcement Learning with Verifiable yet Noisy Rewards under Imperfect Verifiers [90.50039419576807]
  Reinforcement Learning with Verifiable Rewards (RLVR) trains policies against automated verifiers to avoid costly human labeling.<n>To reduce vulnerability to verifier hacking, many RLVR systems collapse rewards to binary $0,1$ during training.<n>This choice carries a cost: it introduces textitfalse negatives (rejecting correct answers, FNs) and textitfalse positives (accepting incorrect ones, FPs)
  arXiv  Detail & Related papers  (2025-10-01T13:56:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.