More Rejective, Not More Discriminative: The Unit of Verification in Pre-Execution LLM Oversight

• URL: http://arxiv.org/abs/2608.23941v1
• Date: Tue, 25 Aug 2026 00:59:55 GMT
• Title: More Rejective, Not More Discriminative: The Unit of Verification in Pre-Execution LLM Oversight
• Abstract summary: Pre-execution oversight is core to trusted monitoring in AI control.<n>Existing designs take the unit as given; its effect on fallible monitors is unmeasured.<n>Our framework is the first controlled, pre-registered instrument for this choice and never reads catch alone.
• Score: 3.8564927363194754
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Pre-execution oversight is core to trusted monitoring in AI control: a fallible LLM monitor vets planned actions before irreversible execution. Over-blocking forfeits usefulness and pressures deployers to disable it. Every protocol must fix a unit of verification: how many actions one call reviews. Existing designs take the unit as given; its effect on fallible monitors is unmeasured. Natural traces cannot isolate it: review length co-varies with error type and position. Catch alone misleads: rejecting everything catches everything. Measuring this needs boundary variation alone and a matched clean control. We introduce the twin-prefix framework, which supplies both. Each gold plan yields a prefix with one injected, environment-accepted error and a clean twin differing in one write. Judging each pair at five nested lengths ties verdict changes to the unit alone. Discrimination is scored by pre-registered informedness, catch minus false rejection. Longer review raises catch; false rejection climbs in lockstep. Informedness peaks at one or two actions for all six judges in both domains: longer windows make zero-shot monitors more rejective, not more discriminative. Replaying withheld observations traces the failure largely to observation deprivation. Safety cases should state the unit and co-report the clean series. Our framework is the first controlled, pre-registered instrument for this choice and never reads catch alone. Our calibrated short unit recovers up to 0.95 informedness over eight-action review, and no tested label-blind policy consistently beats it.

Related papers

• Why LLM Agents Collapse Without Oversight: The Enforcement Gap as the Mechanism Behind Emergence World Failures [19.162716475205308]
  Reflexion-style agents already detect dangerous plan steps through iterative self-critique.<n>We call this the enforcement gap: the audit sees the problem; the controller ignores it.<n>We prove formally that when enforcement probability is near zero, detection quality is irrelevant to security.
  arXiv  Detail & Related papers  (2026-09-14T09:46:59Z)
• IBIB: A Protocol for Measuring Enterprise AI Systems by Serving Route, Not Model Identifier [0.0]
  All 18 audited benchmarks score advertised model identifiers.<n>A gold-blind capability-binding preflight verifies that a route can execute the evaluation contract before any task reaches it.<n>A reliability-inclusive first-pass scoring rule keeps failure in the score while keeping unsupported capability out.
  arXiv  Detail & Related papers  (2026-09-09T17:31:29Z)
• Clean Engineering, Unstable Measurement: A Preregistered Reliability Failure of Black-Box LLM Observers on Shared Endpoints [4.41537705949485]
  Language-model judges now gate training data, score generations, and drive leaderboards.<n>We audited that assumption in two preregistered campaigns with every threshold fixed in advance.<n>Across 52,988 audited request attempts, same-window repeat rankings agreed at Spearman 0.400 against a required 0.90.<n>We distill the evidence into a three-level snapshot-identity ladder, eight design rules, and a reporting checklist.
  arXiv  Detail & Related papers  (2026-09-03T17:59:43Z)
• Bilevel Coordinated Reflection: A Game-Theoretic Approach to Multi-Agent LLM Systems [40.47928908205563]
  Multi-agent LLM systems commonly use an orchestrator to decompose a task for a team of workers and then improve through textual reflection.<n>Despite strong empirical results, these systems lack a unified account of coordination, memory improvement, and the role of external verification.<n>We provide confidence gating for evaluation and re-anchoring guarantees for piecewise-stationary environments.<n>Experiments instantiate these objects with environment-grounded metrics and test the predicted coordination and drift laws.
  arXiv  Detail & Related papers  (2026-09-02T15:50:10Z)
• Safety Does Not Compose: Non-Decaying Loop State for Autonomous LLM Agents [16.24087700490158]
  Large language model agents are increasingly deployed as autonomous loops.<n>We show that this is a failure of composition rather than an implementation detail.<n>We present LoopHarness, which restores a persistent, non-decaying safety state at the loop level.
  arXiv  Detail & Related papers  (2026-08-27T13:52:31Z)
• ADeptS-Bench: Measuring the Trustworthiness of Computer Use Agents Across Devices [48.36933831982682]
  We introduce ADeptS-Bench, a trustworthiness benchmark grounded in the ADEPTS capability framework.<n>The Safety stream provides paired benign/malicious tasks with threats embedded in the visual interface.<n>The Disambiguation stream evaluates whether agents seek clarification when intent is ambiguous.
  arXiv  Detail & Related papers  (2026-08-25T23:21:57Z)
• From Positionwise Confidence to Prefix Scheduling: Verifier Skipping in Speculative Decoding [41.41611499810411]
  Speculative diffusion decoding (SDD) generates every position in a draft block in parallel with a discrete diffusion model.<n>This paper recognizes that this creates a new control handle: whether to invoke the verifier at all.
  arXiv  Detail & Related papers  (2026-08-14T18:00:08Z)
• The LLM Proposes, the Executive Disposes: A Self-Verifying Agent Instrument that Dissociates Commitment Drift from Binding Drift in Long-Horizon Agents [0.0]
  We present an agent instrument built so that verification is structural rather than post-hoc.<n>Every run invalidates itself when per-organ write-error, render-size, or salted-canary-echo floors are breached.<n>We report a clean, single-variable result on a failure every long-horizon agent suffers.
  arXiv  Detail & Related papers  (2026-08-04T15:10:37Z)
• A False Average: Chain-of-Thought Monitors Collapse Where They Are the Only Defense [1.0742675209112622]
  Chain-of-thought (CoT) monitoring is meant to catch the reward hacks that look clean in the actions and betray themselves only in the reasoning.<n>We show that this is exactly where an adversary who controls the reasoning can defeat it.
  arXiv  Detail & Related papers  (2026-08-01T10:42:41Z)
• Zero Hallucination, by Construction: Hallucination-Aware Layered Oversight for Trustworthy Enterprise AI [39.17006317116773]
  We argue that "zero hallucination" is not a property a model possesses but a property a system enforces.<n>We present HALO, an assurance architecture which treats hallucination as a containable failure mode rather than an eliminable one.
  arXiv  Detail & Related papers  (2026-07-20T12:34:44Z)
• AgenticSTS: A Bounded-Memory Testbed for Long-Horizon LLM Agents [34.450981549319266]
  Memory for a long-horizon LLM agent is a contract about what each future decision is allowed to see.<n>We instrument an alternative bounded contract: every decision is made from a fresh user message assembled by typed retrieval.<n>We instantiate the contract in Slay the Spire 2, a closed-rule deck-building game.
  arXiv  Detail & Related papers  (2026-07-02T14:44:32Z)
• What Accuracy and Gradient Cosine Miss: Evaluating Feedback Alignment via Scale Stability, Reference Validity, and Depth Utility [48.10132234701036]
  We propose a diagnostic evaluation protocol based on three checks -- scale stability, reference validity, and depth utility -- together with per-layer rather than aggregate cosine reporting.<n>Across multiple architectures and methods, our protocol identifies all failures with wide calibration margins.
  arXiv  Detail & Related papers  (2026-06-19T06:04:17Z)
• Stateful Online Monitoring Catches Distributed Agent Attacks [63.00394432922707]
  We develop an online stateful monitor that uses real-time clustering to collect weak suspiciousness signals across many agent transcripts.<n>Our results point toward a new class of safety monitors which reason over groups of users rather than isolated transcripts.
  arXiv  Detail & Related papers  (2026-05-29T17:57:00Z)
• A Universal Cliff and a Design Fingerprint: Cross-Section Defect Detection Under LLM Orchestration [0.0]
  Production language-model systems answer a request byspanning it across an invisible orchestration of worker agents.<n>We ask what this does to a class of defect no single worker can see.<n>We vary only the model -- ten systems across five generations from one developer and five providers from distinct alignment paradigms.
  arXiv  Detail & Related papers  (2026-05-25T05:09:48Z)
• Senses Wide Shut: A Representation-Action Gap in Omnimodal LLMs [41.451167369762544]
  We document a Representation-Action Gap: hidden states reliably encode premise-perception mismatches.<n> Behaviorally, models fall into two failure modes: under-rejection and over-rejection.<n>As an initial diagnostic intervention, a probe-guided logit adjustment re-injects the encoded mismatch signal into decoding.
  arXiv  Detail & Related papers  (2026-05-13T16:14:44Z)
• TraceGuard: Structured Multi-Dimensional Monitoring as a Collusion-Resistant Control Protocol [1.0195618602298684]
  TraceGuard is a structured multi-dimensional monitoring protocol that evaluates agent actions across five dimensions.<n>It is implemented as a new monitor type for the open-source ControlArena framework.
  arXiv  Detail & Related papers  (2026-04-05T05:05:59Z)
• CoVerRL: Breaking the Consensus Trap in Label-Free Reasoning via Generator-Verifier Co-Evolution [52.691495954442985]
  CoVerRL is a framework where a single model alternates between generator and verifier roles, with each capability bootstrapping the other.<n> Experiments across Qwen and Llama model families demonstrate that CoVerRL outperforms label-free baselines by 4.7-5.9% on mathematical reasoning benchmarks.<n>Self-verification accuracy improves from around 55% to over 85%, confirming that both capabilities genuinely co-evolve.
  arXiv  Detail & Related papers  (2026-03-18T14:38:55Z)
• When Safety Blocks Sense: Measuring Semantic Confusion in LLM Refusals [0.07777489763207261]
  Current evaluations mostly report global rates such as false rejection or compliance.<n>We introduce "semantic confusion," a failure mode that captures such local inconsistency.<n>We propose three model-agnostic metrics at the token level: Confusion Index, Confusion Rate, and Confusion Depth.
  arXiv  Detail & Related papers  (2025-11-30T19:11:45Z)
• A Granular Study of Safety Pretraining under Model Abliteration [64.24346997570275]
  We study model abliteration, a lightweight projection technique designed to remove refusal-sensitive directions.<n>We issue 100 prompts with balanced harmful and harmless cases, classify responses as **Refusal** or **Non-Refusal** using multiple judges, and validate judge fidelity.<n>Our study produces a checkpoint-level characterization of which data-centric safety components remain robust under abliteration.
  arXiv  Detail & Related papers  (2025-10-03T07:01:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.