A False Average: Chain-of-Thought Monitors Collapse Where They Are the Only Defense

• URL: http://arxiv.org/abs/2608.00583v1
• Date: Sat, 01 Aug 2026 10:42:41 GMT
• Title: A False Average: Chain-of-Thought Monitors Collapse Where They Are the Only Defense
• Abstract summary: Chain-of-thought (CoT) monitoring is meant to catch the reward hacks that look clean in the actions and betray themselves only in the reasoning.<n>We show that this is exactly where an adversary who controls the reasoning can defeat it.
• Score: 1.0742675209112622
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Chain-of-thought (CoT) monitoring is meant to catch the reward hacks that look clean in the actions and betray themselves only in the reasoning. We show that this is exactly where an adversary who controls the reasoning can defeat it. Rewriting only an agent's reasoning to read as good-faith engineering, while copying every command and output verbatim so the exploit is unchanged, drops a held-out monitor's catch rate on that subset from about 95% to under 11% in one gradient-free shot. A monitor's aggregate accuracy is a false average: dominated by hacks the actions give away, it hides the near-total collapse this rewrite produces on the subset where CoT monitoring is the only signal. The attack transfers across monitor families and agent models, reproduces with live agents, though against a calibrated monitor evasion concentrates in the strongest agent. Trace-only defenses recover it only partially, even one primed on the attack, because the rewrite stays truthful about what happened and lies only about intent; only information from outside the trace helps substantially. A probe on an open-weight surrogate monitor's activations separates the hacks its verdict misses, but a causal control shows this is a detector, not evidence the monitor secretly knows.

Related papers

• More Rejective, Not More Discriminative: The Unit of Verification in Pre-Execution LLM Oversight [3.8564927363194754]
  Pre-execution oversight is core to trusted monitoring in AI control.<n>Existing designs take the unit as given; its effect on fallible monitors is unmeasured.<n>Our framework is the first controlled, pre-registered instrument for this choice and never reads catch alone.
  arXiv  Detail & Related papers  (2026-08-25T00:59:55Z)
• When Local Monitors Miss Compositional Harm: Diagnosing Distributed Backdoors in Multi-Agent Systems [4.695509706472705]
  Local safety is not global safety when harm is compositional.<n>Local monitors lose the signal exactly as local evidence disappears, and it returns only when the monitor sees the assembled object.<n>Full-trace monitors and decoders still fail unless they reach the representation where the payload is exposed.
  arXiv  Detail & Related papers  (2026-07-13T16:08:46Z)
• Calibration-Family Overfit: Why Trusted Sabotage Monitors Don't Transfer Across Lineages [0.7252027234425333]
  We make the untrusted policy family the controlled axis and decompose the cross-family AUROC into how obvious each family's sabotage is.<n>On code-backdoor transcripts the interaction is positive and survives the dominant confounds.<n>We give a four-step protocol that acts on the gap, with each step a measured result.
  arXiv  Detail & Related papers  (2026-07-06T22:40:07Z)
• Stateful Online Monitoring Catches Distributed Agent Attacks [63.00394432922707]
  We develop an online stateful monitor that uses real-time clustering to collect weak suspiciousness signals across many agent transcripts.<n>Our results point toward a new class of safety monitors which reason over groups of users rather than isolated transcripts.
  arXiv  Detail & Related papers  (2026-05-29T17:57:00Z)
• SLEIGHT-Bench: A Benchmark of Evasion Attacks Against Agent Monitors [3.989565392866092]
  We present SLEIGHT-Bench (Subtle Low-itEration Insight-Guided Harmful Transcripts), a benchmark of synthetic transcripts containing 40 attacks across 11 categories.<n>We find that 20 of 40 attacks are never caught by an Opus 4.6 monitor with extended thinking across 10 trials at a 1% false-positive rate, while the overall catch rate is 32%.
  arXiv  Detail & Related papers  (2026-05-15T20:46:46Z)
• Do Synthetic Trajectories Reflect Real Reward Hacking? A Systematic Study on Monitoring In-the-Wild Hacking in Code Generation [53.024513172383195]
  We present a systematic analysis of the synthetic vs. in-the-wild discrepancy in reward hacking.<n>We examine what extent hacking behaviors induced by prompting resemble those emerging during RL training.<n>We find that synthetic-data-trained monitors fail to generalize to "in-the-wild" hacking.
  arXiv  Detail & Related papers  (2026-04-26T01:26:50Z)
• Noticing the Watcher: LLM Agents Can Infer CoT Monitoring from Blocking Feedback [48.40096834514452]
  Chain-of-thought (CoT) monitoring is proposed as a method for overseeing the internal reasoning of language-model agents.<n>We ask whether reasoning agents can autonomously infer that their supposedly private CoT is under surveillance.<n>We find that frontier models can deduce the existence of this monitor purely from blocking feedback.
  arXiv  Detail & Related papers  (2026-03-14T00:12:14Z)
• Detecting Object Tracking Failure via Sequential Hypothesis Testing [80.7891291021747]
  Real-time online object tracking in videos constitutes a core task in computer vision.<n>We propose interpreting object tracking as a sequential hypothesis test, wherein evidence for or against tracking failures is gradually accumulated over time.<n>We propose both supervised and unsupervised variants by leveraging either ground-truth or solely internal tracking information.
  arXiv  Detail & Related papers  (2026-02-13T14:57:15Z)
• Adaptive Attacks on Trusted Monitors Subvert AI Control Protocols [80.68060125494645]
  We study adaptive attacks by an untrusted model that knows the protocol and the monitor model.<n>We instantiate a simple adaptive attack vector by which the attacker embeds publicly known or zero-shot prompt injections in the model outputs.
  arXiv  Detail & Related papers  (2025-10-10T15:12:44Z)
• CoT Red-Handed: Stress Testing Chain-of-Thought Monitoring [3.6284577335311563]
  Chain-of-Thought (CoT) monitoring improves detection by up to 27 percentage points in scenarios where action-only monitoring fails to reliably identify sabotage.<n>CoT traces can also contain misleading rationalizations that deceive the monitor, reducing performance in more obvious sabotage cases.<n>This hybrid monitor consistently outperforms both CoT and action-only monitors across all tested models and tasks, with detection rates over four times higher than action-only monitoring for subtle deception scenarios.
  arXiv  Detail & Related papers  (2025-05-29T15:47:36Z)
• Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation [56.102976602468615]
  We show that we can monitor a frontier reasoning model, such as OpenAI o3-mini, for reward hacking in agentic coding environments.<n>We find that with too much optimization, agents learn obfuscated reward hacking, hiding their intent within the chain-of-thought.
  arXiv  Detail & Related papers  (2025-03-14T23:50:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.