Auditing CoT Answer-Hijack Patches: Source-Control Certificates with Type-I Guarantees

• URL: http://arxiv.org/abs/2606.04717v2
• Date: Wed, 10 Jun 2026 09:03:21 GMT
• Title: Auditing CoT Answer-Hijack Patches: Source-Control Certificates with Type-I Guarantees
• Abstract summary: Chain-of-thought (CoT) answer-hijack templates can flip the final numeric answer of a 7B-8B language model on GSM8K or MATH-500.<n>We show that a successful clean-source patch is often read as evidence that the patched activation carries the recovered content.<n>We construct an audit that turns each candidate patch into a source-control certificate with a pre-registered Type-I guarantee.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Chain-of-thought (CoT) answer-hijack templates can flip the final numeric answer of a 7B-8B language model on GSM8K or MATH-500 even when the visible reasoning trace looks fluent. Activation patching is the standard probe for locating where this hijack can be undone, and a successful clean-source patch is often read as evidence that the patched activation carries the recovered content. We show that this reading is unsound: clean-only localization profiles (peak, spread, thresholded band) underidentify the frozen-hook source contrast, and the clean-only profile is an intervention map, not a mediation certificate. We then construct an audit that turns each candidate patch into a source-control certificate with a pre-registered Type-I guarantee. The certificate runs in three stages: SELECT (clean-source band sweep with permutation calibration and held-out validation), FREEZE (lock the hook), and AUDIT (paired-bootstrap source contrasts at the frozen hook). It emits an incorrect mechanism label with probability at most alpha = alpha_sel + alpha_audit under sample-split disjointness. A matching-rate sample-complexity theorem (n_star = Theta(Delta^{-2} log(1/alpha))) bounds the audit cost. On Qwen2.5-7B and Llama3-8B, three few-shot/puzzle cells pass confirmatory K=1 localization with held-out gaps +32.6, +45.1, +17.7; fixed-hook reruns recover 47.0% (Qwen-puzzle) and 39.0% (Llama3-puzzle) at n=100; frozen MATH-500 transfer recovers 26.0%. After audit, Llama3-PZ and Qwen-PZ are identity-light with moderate magnitude (Qwen-PZ also layer-sensitive); Llama3-FS is a single-seed moderate-positive candidate (multi-seed replication queued); Qwen-FS is exploratory non-separation with a layer-sensitive flag. The method is a diagnostic auditing protocol, not an adaptive safety defense.

Related papers

• BeTaL-GBI: Admission-Aware Benchmark Tuning and Full-Stack Verification of Geometric Belief Interfaces [2.894286977279531]
  This study examines whether an enterprise verification architecture can isolate interface failure, task competence, policy admissibility, and control integrity while keeping claims auditable.<n>BeTaL-GBI v0.2 applies Benchmark Tuning with an LLM-in-the-loop over 2,218,750,380 grid points, separating format admission from conditional performance.<n>Across 512 synthetic tasks, a 16-gate policy detects all 116 injected severe contradictions and accepts all 99 clean records. Hallucinator and evidence-forger surrogates are blocked with zero silent promotions.
  arXiv  Detail & Related papers  (2026-08-21T16:52:38Z)
• The Recall Trap: A Recall-Maximizing Retriever Configuration Reduces Issue Resolution in Fixed-Budget Code Context [45.88028371034407]
  We report a controlled case study in code repair, not a new phenomenon.<n>We toggle one flag (one-chunk-perfile deduplication) on an otherwise identical stack.<n>We map where it holds: it reverses on a lexical BM25 (3.2pp, significant cross-paradigm interaction)<n>We show that at a tight fixed budget, do not hard-deduplicate by file, and A/B packing policies against the task, not the metric flag was tuned.
  arXiv  Detail & Related papers  (2026-08-14T19:22:50Z)
• From Positionwise Confidence to Prefix Scheduling: Verifier Skipping in Speculative Decoding [41.41611499810411]
  Speculative diffusion decoding (SDD) generates every position in a draft block in parallel with a discrete diffusion model.<n>This paper recognizes that this creates a new control handle: whether to invoke the verifier at all.
  arXiv  Detail & Related papers  (2026-08-14T18:00:08Z)
• Beyond Local Accuracy: A Protocol-Level Identifiability Audit for Controlled LLM Reasoning Evaluation [6.047519836191728]
  We formalize a protocol-level identifiability audit over a finite behavioral policy class.<n>The audit requires zero model calls and resolves our diagnostic case.<n>This case shows how evaluation-design validity can be checked structurally before model inference.
  arXiv  Detail & Related papers  (2026-08-13T14:49:47Z)
• When Many Answers Are Valid, Voting Fails: Symbolic Verification for Best-of-K Causal Reasoning in LLMs [42.32694162421078]
  We introduce CALVER, a training-free symbolic verifier that scores structured traces against Pearl's causal criteria.<n>On CLEAR find-one-valid queries that admit multiple graph-valid answers, CALVER reaches 42.1% where plurality, a reward model, an LLM judge, and model confidence remain near 30%.
  arXiv  Detail & Related papers  (2026-08-04T11:45:46Z)
• Valid Per-Field Selective Risk Control for Document Extraction: Three Failure Modes, a Validity Ladder, and When Conditioning Pays [0.0]
  Per-field accept/review with selective risk at most alpha is the trust contract document-extraction systems need.<n>On 13,859 genuine claude-sonnet-5 fields from 800 CORD receipts (49.0% correct) we diagnose three failure modes.<n>Released Apache-2.0 with seed-pinned, regression-gated procedures.
  arXiv  Detail & Related papers  (2026-07-28T17:08:50Z)
• Looping Is Not Reliability: State-Bound Evidence and Typed Revision Contracts for Agentic Code Repair [36.56438114281786]
  We study the gap between finding a correct patch and retaining, verifying, and submitting it.<n>We derive an evidence-bound typed loop contract and instantiate its mechanically enforceable subset in a reference implementation.
  arXiv  Detail & Related papers  (2026-07-27T16:05:23Z)
• Right Makes Might: Aligning Verified Hidden States Empowers RL Reasoning [55.264863369127774]
  Current methods reduce each correct rollout to a single reward bit, ignoring the geometric structure shared among their hidden states.<n>We propose Hidden-Align, an auxiliary loss function that aligns the last-layer hidden states of correct rollouts at the anchor token during RL training, with zero overhead in both training and inference.<n>On eight mathematical reasoning benchmarks, Hidden-Align improves average pass@1 over the DAPO baseline by 3.8, 6.2, and 5.4 percentage points on Qwen3-1.7B, 4B, and 14B respectively, with consistent pass@k gains across all three scales.
  arXiv  Detail & Related papers  (2026-06-02T06:51:15Z)
• REPOT: Recoverable Program-of-Thought via Checkpoint Repair [0.03038255798961237]
  One-shot Program-of-Thought (PoT) emits a Python program that prints a primitive-action plan.<n>We introduce RePoT: a deterministic verified replay that walks the plan through the environment to its first invalid transition.<n>RePoT beats PoT by +3 to +11pp across four closed-model configurations.
  arXiv  Detail & Related papers  (2026-05-28T15:03:17Z)
• The Chain Holds, the Answer Folds: Trace-Answer Dissociation in Reasoning Models Under Adversarial Pressure [8.86745721473138]
  Reasoning models are evaluated on single-turn benchmarks but deployed in multi-turn dialogue, where users push back on correct answers.<n>We call this unfaithful capitulation (UC) and isolate it with a $2times 2$ latent-versus-behavioral framework that flip-rate metrics and single-turn probes both miss.
  arXiv  Detail & Related papers  (2026-05-27T20:41:08Z)
• What Makes a Medical Checker Trainable? Diagnosing Signal Collapse and Reward Hacking in Checker-Guided RAG for Biomedical QA [6.163604976615509]
  Medical RAG needs evidence-grounded claims, so plugging a claim-level NLI checker into retrieval-augmented RL is intuitive.<n>We compare four NLI checker back-ends as process rewards inside a GRPO-trained medical RAG agent.
  arXiv  Detail & Related papers  (2026-05-25T16:06:06Z)
• Constraint-Anchored Attribution: Feasibility-Certified Counterfactuals and Bonferroni-PAC Sufficient Subsets for Neural CO Policies [0.0]
  We give an attribution method for neural-optimisation (CO) policies that decomposes a decision by constraint families via LP-relaxation duals.<n>Across three CO problems and three seeds, our LP-anchored $$-attribution matches the CF-derived signal.
  arXiv  Detail & Related papers  (2026-05-24T19:46:30Z)
• Distributional Energy-Based Models for Uncertainty-Aware Structured LLM Reasoning [40.342912574072024]
  Large Language Models produce structured outputs such as travel plans or code solutions.<n>Individual reasoning steps may appear correct while the output as a whole violates budgets, fails test cases, or contradicts earlier deductions.<n>We propose a learned quality scorer with deterministic analytical constraint penalties for verifying structured LLM outputs.
  arXiv  Detail & Related papers  (2026-05-15T17:08:27Z)
• The Last Word Often Wins: A Format Confound in Chain-of-Thought Corruption Studies [0.0]
  Corruption studies infer which steps are computationally important'' from accuracy loss when steps are corrupted.<n>We show that when benchmark chains end with an explicit terminal answer line, these tests largely measure emphanswer placement rather than where intermediate computation is carried out.
  arXiv  Detail & Related papers  (2026-05-11T16:26:50Z)
• SURE-RAG: Sufficiency and Uncertainty-Aware Evidence Verification for Selective Retrieval-Augmented Generation [6.604874054866016]
  We present SURE-RAG, a transparent aggregation protocol built on the observation that evidence sufficiency is a set-level property.<n>A shared pair-level claim-evidence verifier produces local relation distributions, which SURE-RAG aggregates into interpretable answer-level signals.<n>We evaluate on HotpotQA-RAG v3, a controlled multi-hop benchmark, under an artifact-aware protocol.
  arXiv  Detail & Related papers  (2026-05-05T09:05:40Z)
• ZenBrain: A Neuroscience-Inspired 7-Layer Memory Architecture for Autonomous AI Systems [51.56484100374058]
  On LongMemEval-500, ZenBrain matches a long-context oracle's binary-judge accuracy to within 4.5 pp.<n> ZenBrain is a 7-layer neuroscience-inspired memory architecture.
  arXiv  Detail & Related papers  (2026-04-26T20:39:19Z)
• Correction and Corruption: A Two-Rate View of Error Flow in LLM Protocols [51.56484100374058]
  We propose a paired-outcome measurement interface for auditing a single protocol step on exact-match tasks.<n>For each instance, the interface records a baseline correctness bit and a post-step correctness bit.<n>These rates predict accuracy changes and define a reusable empirical interface testable across seeds, mixtures, and pipelines.
  arXiv  Detail & Related papers  (2026-04-20T13:25:40Z)
• Safety Recovery in Reasoning Models Is Only a Few Early Steering Steps Away [97.11976870616273]
  We propose a lightweight inference-time defense that treats safety recovery as a satising constraint rather than an objective.<n>In our evaluations across six open-source MLRMs and four jailbreak benchmarks, SafeThink reduces attack success rates by 30-60%.
  arXiv  Detail & Related papers  (2026-02-11T18:09:17Z)
• Communication-Constrained Bandits under Additive Gaussian Noise [111.06688156723018]
  We study a distributed multi-armed bandit where a client supplies the learner with communication-constrained feedback. We propose a multi-phase bandit algorithm, $mathttUEtext-UCB++$, which matches this lower bound to a minor additive factor.
  arXiv  Detail & Related papers  (2023-04-25T09:31:20Z)
• Faithful Chain-of-Thought Reasoning [51.21714389639417]
  Chain-of-Thought (CoT) prompting boosts Language Models' (LM) performance on a gamut of reasoning tasks. We propose Faithful CoT, a reasoning framework involving two stages: Translation and Problem Solving. This guarantees that the reasoning chain provides a faithful explanation of the final answer.
  arXiv  Detail & Related papers  (2023-01-31T03:04:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.