Prior Audit-Repair Context Shifts LLM Verifier Thresholds Toward Leniency

• URL: http://arxiv.org/abs/2608.16003v1
• Date: Mon, 17 Aug 2026 01:41:43 GMT
• Title: Prior Audit-Repair Context Shifts LLM Verifier Thresholds Toward Leniency
• Abstract summary: We measure false alarms on human-verified-correct ProcessBench traces with the present task held byte-identical.<n>We find that a completed audit -> repair episode already in the model's context lowers false alarms in 15 of 15 model x wording combinations.
• Score: 0.025718125188898048
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Automated checking pipelines increasingly place one language model as the checker and another (or the same one) as the fixer. We ask whether that wiring changes what the checker reports. Measuring false alarms on human-verified-correct ProcessBench traces with the present task held byte-identical, we find that a completed audit -> repair episode already in the model's context lowers false alarms in 15 of 15 model x wording combinations, by 2.8 to 11.5 percentage points against a length-matched non-audit control, a 9 to 25% reduction relative to that control. The direction contradicts what the accumulated-message literature predicts: an episode whose audit reported an error lowers false alarms further still, at all five wordings on the model where that manipulation lands cleanly, though a negativity asymmetry predicts more flagging. Decomposing the episode finds repair content and audit verdict complementary: different components carry the effect on different model families. Signal-detection analysis locates the change in the threshold rather than in discrimination -- the criterion moves in 15 of 15 combinations and survives correction in 13 while d' survives in none, though the d' test is half as sensitive by construction -- and a hand audit of 50 false alarms finds 82% simply wrong, so at this operating point the shift need not be harmful. With reasoning enabled the effect keeps its relative size on both models tested, and the threshold reading holds there too.

Related papers

• Detecting Safety Training Modification in Language Models via Activation Analysis [0.0]
  We introduce AMS, a tool that detects modifications to safety training in language models by measuring the geometric structure of safety-relevant concepts in activation space.<n>We validate AMS across 14 model configurations spanning 4 architecture families.<n>Mechanistic analysis identifies a four-class taxonomy of safety-training modifications distinguished by activation-space signature.
  arXiv  Detail & Related papers  (2026-08-06T03:57:38Z)
• Looping Is Not Reliability: State-Bound Evidence and Typed Revision Contracts for Agentic Code Repair [36.56438114281786]
  We study the gap between finding a correct patch and retaining, verifying, and submitting it.<n>We derive an evidence-bound typed loop contract and instantiate its mechanically enforceable subset in a reference implementation.
  arXiv  Detail & Related papers  (2026-07-27T16:05:23Z)
• CUSUM-Shaped Inference-Time Monitoring and Targeted Re-Decoding for Quantized Small Language Model Reasoning [0.6999740786886536]
  MGT-B maps overlapping windows of pre-sampling uncertainty and degeneration features to position-conditional empirical tail probabilities.<n>We audit whether the effect persists on problem identities first observed after the manual choice of log threshold h = 10.<n>Neither analysis is confirmatory, and the empirical factors are not established as a valid e-process or e-detector.
  arXiv  Detail & Related papers  (2026-07-22T13:34:17Z)
• Catching One in Five: LLM-as-Judge Blind Spots in Production Multi-Turn Transaction Agents [45.148328075418156]
  We study a deployed multi-turn food-and-beverage ordering agent and measure how many genuine quality problems it catches.<n>Our blind-spot taxonomy shows the failure is structured, not random.<n>For production multi-turn agents, automated judging is a regression floor, not a substitute for human review.
  arXiv  Detail & Related papers  (2026-06-09T02:11:01Z)
• Distributional Energy-Based Models for Uncertainty-Aware Structured LLM Reasoning [40.342912574072024]
  Large Language Models produce structured outputs such as travel plans or code solutions.<n>Individual reasoning steps may appear correct while the output as a whole violates budgets, fails test cases, or contradicts earlier deductions.<n>We propose a learned quality scorer with deterministic analytical constraint penalties for verifying structured LLM outputs.
  arXiv  Detail & Related papers  (2026-05-15T17:08:27Z)
• Silent Commitment Failure in Instruction-Tuned Language Models: Evidence of Governability Divergence Across Architectures [0.0]
  We introduce governability -- the degree to which a model's errors are detectable before output commitment and correctable once detected.<n>We show benchmark accuracy does not predict governability, correction capacity varies independently of detection, and identical governance scaffolds produce opposite effects across models.<n>We propose a Detection and Correction Matrix classifying model-task combinations into four regimes: Governable, Monitor Only, Steer Blind, and Ungovernable.
  arXiv  Detail & Related papers  (2026-03-22T21:50:28Z)
• HLE-Verified: A Systematic Verification and Structured Revision of Humanity's Last Exam [63.84155758655084]
  Humanity's Last Exam (HLE) has become a widely used benchmark for evaluating frontier large language models.<n>We introduce HLE-Verified, a verified and revised version of HLE with a transparent verification protocol and fine-grained error taxonomy.<n>We evaluate seven state-of-the-art language models on HLE and HLE-Verified, observing an average absolute accuracy gain of 7--10 percentage points.
  arXiv  Detail & Related papers  (2026-02-15T02:50:15Z)
• SpatialBench-UC: Uncertainty-Aware Evaluation of Spatial Prompt Following in Text-to-Image Generation [0.0]
  SpatialBench-UC is a small, reproducible benchmark for pairwise spatial relations.<n>We release a benchmark package, versioned prompts, pinned configs, per-sample checker outputs, and report tables.<n>We evaluate three baselines, Stable Diffusion 1.5, SD 1.5 BoxDiff, and SD 1.4 GLIGEN.
  arXiv  Detail & Related papers  (2026-01-19T23:37:10Z)
• Detecting Adversarial Fine-tuning with Auditing Agents [38.964973163076586]
  We introduce the concept of a fine-tuning auditing agent and show it can detect harmful fine-tuning prior to model deployment.<n>We evaluate our detection approach on a diverse set of eight strong fine-tuning attacks from the literature, along with five benign fine-tuned models.<n>Most promising, the auditor is able to detect covert cipher attacks that evade safety evaluations and content moderation of the dataset.
  arXiv  Detail & Related papers  (2025-10-17T23:01:16Z)
• How Sampling Affects the Detectability of Machine-written texts: A Comprehensive Study [39.866323800060066]
  Large Language Models (LLMs) are increasingly common and often indistinguishable from human-written content.<n>Many recent detectors report near-perfect accuracy, often boasting AUROC scores above 99%.<n>In this work, we examine how sampling-based decoding impacts detectability.
  arXiv  Detail & Related papers  (2025-10-15T15:36:45Z)
• A Granular Study of Safety Pretraining under Model Abliteration [64.24346997570275]
  We study model abliteration, a lightweight projection technique designed to remove refusal-sensitive directions.<n>We issue 100 prompts with balanced harmful and harmless cases, classify responses as **Refusal** or **Non-Refusal** using multiple judges, and validate judge fidelity.<n>Our study produces a checkpoint-level characterization of which data-centric safety components remain robust under abliteration.
  arXiv  Detail & Related papers  (2025-10-03T07:01:45Z)
• Causal Understanding by LLMs: The Role of Uncertainty [43.87879175532034]
  Recent papers show LLMs achieve near-random accuracy in causal relation classification.<n>We investigate whether pretraining exposure to causal examples improves causal understanding.
  arXiv  Detail & Related papers  (2025-09-24T13:06:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.