What Do Compliance Detectors Read? An Audit of Activation Probes and Guard Models

• URL: http://arxiv.org/abs/2608.16852v1
• Date: Mon, 17 Aug 2026 17:37:07 GMT
• Title: What Do Compliance Detectors Read? An Audit of Activation Probes and Guard Models
• Abstract summary: Internal Compliance Score (ICS) is a training-free activation readout from ten labelled pairs and scored by a single projection.<n>We show this condition fails across the current class of compliance detectors, a failure we call rule blindness.<n>We release the counterfactual protocol and crossed-rule benchmark so rule blindness can be tested in future probe and guard claims.
• Score: 3.3598755777055374
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Regulatory compliance monitoring in deployed language models is increasingly implemented as a legal and audit control, checking model outputs against written rules spanning data protection, healthcare, financial regulation, and platform policy. Such monitoring is meaningful only if a detector's verdict depends on the stated rule rather than on surface features of the scenario. We show this condition fails across the current class of compliance detectors, a failure we call rule blindness. Deleting, permuting, or substituting the governing rule leaves detection accuracy unchanged for every guard and activation probe we test, including a policy-conditioned guard that correctly cites the governing clause yet barely changes its verdict when that clause is swapped for its permissive counterpart. A purpose-built benchmark crossing two rules with two scenarios, so that neither alone predicts the label, confirms the failure under a design no prior benchmark rules out, and shows that step by step reasoning, not any fast detector we test, is what escapes it. Auditing at scale requires a retraining-free detector, so we introduce the Internal Compliance Score (ICS): a training-free activation readout calibrated from ten labelled pairs and scored by a single projection. We hold ICS to the same scrutiny as the guards it audits: a pre-registered criterion for beating trivial baselines is not met, and a bag-of-words model matches its pooled generalisation exactly. It remains useful because it is inexpensive, letting us audit four deployed guard models, an 8B zero-shot judge, and thirteen benchmarks, and it raises the mechanically verified pass rate when used to rank candidate responses, though an adaptive white-box attack removes this gain. We release the counterfactual protocol and crossed-rule benchmark so rule blindness can be tested in future probe and guard claims.

Related papers

• From Subjective Judgments to Auditable Standards:Protocol-Guided AI Auditing of Website Redundancy [2.2917707112773593]
  Website redundancy does not have a single fixed meaning.<n>We introduce CORA, which measures repetition load, normal-use tax, and failure-domain recovery reserve separately.
  arXiv  Detail & Related papers  (2026-08-21T07:17:35Z)
• SynthGuard-ReleaseBench: Locked-Audit Evidence for Synthetic Tabular Data Releases [0.0]
  We introduce SynthGuard-ReleaseBench, an audit framework that locks the use, candidate panel, tolerances, and audit schedule before evaluation.<n>It compares real-trained and synthetic-trained on protected data, gives simultaneous finite-sample bounds for loss gaps, requires controls, and keeps utility, empirical privacy risk mechanism claims separate.
  arXiv  Detail & Related papers  (2026-08-14T03:12:17Z)
• SVR: Self-Verifying Refinement via Joint Verdict-Confidence Reinforcement Learning for Adaptive Test-Time Compute [62.3458279176813]
  Self-Verifying Refinement is an oracle-free multi-turn reinforcement learning framework.<n>It learns to use self-verification as a compute-control policy.<n>It achieves a macro-average accuracy of 0.563 with only 2.99 inference turns on average.
  arXiv  Detail & Related papers  (2026-07-30T16:20:58Z)
• What Can Be Enforced? A Theory of Certified Runtime Safety for Tool-Using Agents [0.0]
  guardrails act before irreversible tool calls, but their guarantees depend on what policy state is representable.<n>Pearson representation attacks add a margin, so benign calibration alone does not transfer.<n>Experiments target these distinctions through static diagnostics, controlled-model enumeration, representation rewrites, and paired closed-loop reruns.
  arXiv  Detail & Related papers  (2026-07-24T19:14:26Z)
• Evaluating and Guarding Citation Faithfulness in Agentic Scientific Synthesis [2.7092559671391]
  Agentic LLM systems such as OpenScholar and PaperQA2 read the scientific literature and return cited answers.<n>Neither audits the reliability of that check itself.<n>We show it is not reliable, and that this matters.<n>We present a gold-anchored evaluation protocol and a deployable guard.
  arXiv  Detail & Related papers  (2026-07-10T02:05:17Z)
• Verification of Adaptive Agentic Controllers through Finite Rule Revision [0.0]
  This paper formulates a bounded verification protocol for adaptive agentic controllers represented by finite symbolic rules.<n> diagnostic failures are mapped to predefined rule-level edits, including rule addition, rule deletion, and priority revision.<n> repaired controllers are then evaluated on held-out simulation seeds or cloned initial states.
  arXiv  Detail & Related papers  (2026-07-07T13:38:01Z)
• Bayesian control for coding agents [63.64172141184361]
  We propose a cost-sensitive sequential hypothesis testing framework for coding agents.<n>A Bayesian controller maintains a belief over correctness and decides whether to gather more evidence, refine the candidate, verify it, or stop.<n>We show that the belief state yields an interpretable correctness score that outperforms token-probability and raw tool-success baselines for uncertainty quantification.
  arXiv  Detail & Related papers  (2026-06-23T11:41:32Z)
• CIAware-Bench: Benchmarking Control Intervention Awareness Across Frontier LLMs [100.38986535324284]
  We introduce textbfCIAware-Bench, a benchmark for measuring textbfcontrol textbfintervention (CI) awareness across frontier models.<n>CIAware-Bench tests whether models can distinguish their own trajectories from those modified by a control intervention.
  arXiv  Detail & Related papers  (2026-06-09T16:24:16Z)
• The Hidden Signal of Verifier Strictness: Controlling and Improving Step-Wise Verification via Selective Latent Steering [67.8271652641864]
  We study whether verifier strictness can be controlled through hidden-state intervention.<n>VerifySteer exploits latent correctness signals for sample-level routing and selectively intervenes on paragraph boundaries.
  arXiv  Detail & Related papers  (2026-05-20T05:48:16Z)
• When No Benchmark Exists: Validating Comparative LLM Safety Scoring Without Ground-Truth Labels [34.86529553336423]
  Many deployments must compare candidate language models for safety before a labeled benchmark exists for the relevant language, sector, or regulatory regime.<n>We formalize this setting as benchmarkless comparative safety scoring and specify the contract under which a scenario-based audit can be interpreted as deployment evidence.<n>Scores are valid only under a fixed scenario pack, rubric, auditor, judge, sampling configuration, and rerun budget.
  arXiv  Detail & Related papers  (2026-05-07T17:56:41Z)
• A Granular Study of Safety Pretraining under Model Abliteration [64.24346997570275]
  We study model abliteration, a lightweight projection technique designed to remove refusal-sensitive directions.<n>We issue 100 prompts with balanced harmful and harmless cases, classify responses as **Refusal** or **Non-Refusal** using multiple judges, and validate judge fidelity.<n>Our study produces a checkpoint-level characterization of which data-centric safety components remain robust under abliteration.
  arXiv  Detail & Related papers  (2025-10-03T07:01:45Z)
• Auditing AI models for Verified Deployment under Semantic Specifications [65.12401653917838]
  AuditAI bridges the gap between interpretable formal verification and scalability. We show how AuditAI allows us to obtain controlled variations for verification and certified training while addressing the limitations of verifying using only pixel-space perturbations.
  arXiv  Detail & Related papers  (2021-09-25T22:53:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.