Behavior Cue Reasoning: Monitorable Reasoning Improves Efficiency and Safety through Oversight

• URL: http://arxiv.org/abs/2605.07021v1
• Date: Thu, 07 May 2026 23:05:50 GMT
• Title: Behavior Cue Reasoning: Monitorable Reasoning Improves Efficiency and Safety through Oversight
• Abstract summary: Reasoning in Large Language Models (LLMs) poses a challenge for oversight.<n>We introduce Behavior Cue Reasoning for making LLM reasoning more controllable and monitorable.
• Score: 8.531991753482838
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reasoning in Large Language Models (LLMs) poses a challenge for oversight as many misaligned behaviors do not surface until reasoning concludes. To address this, we introduce Behavior Cue Reasoning for making LLM reasoning more controllable and monitorable. Behavior Cues are special token sequences that a model is trained to emit immediately before specific implicit and explicit behaviors, acting as dual purpose signal and control levers. When fine-tuning a weaker external monitor with Reinforcement Learning for reasoning oversight, a compressed view of only information surfaced by Behavior Cues is sufficient signal for the monitor to prune up to 50% of otherwise wasted reasoning tokens in complex math problem solving. When leveraged by an almost optimal rule-based monitor in an environment where excessive constraint violations results in failure, \ours allows for the recovery of safe actions from 80% of reasoning traces that would otherwise end with the proposal of an unsafe action, more than doubling the success rate from 46% to 96%. Through evaluation across two model families and three domains, we show that \bcreasoning improves reasoning monitorability and controllability with no cost to performance. More broadly, our work progresses scalable oversight by demonstrating how the monitored model itself can be trained to reason more tractably to oversight. Code to be released at https://github.com/christopherzc/text-games

Related papers

• Does Out-of-Sight Equal Out-of-Mind in CoT Monitorability? [18.08925355421489]
  Chain-of-thought (CoT) reasoning offers a window into the decision-making of large language models.<n>Latent CoT approaches replace explicit tokens with a small number of continuous states, lowering inference costs but removing the readable trace this monitoring relies on.<n>We study this question with a hint-based intervention setup, a proxy for behaviors where models exploit biasing input cues.
  arXiv  Detail & Related papers  (2026-08-05T14:55:23Z)
• Knowing When to Quit: Diagnosing and Training LLMs to Abort Futile Reasoning [109.33717747526583]
  Large language models generate computationally expensive yet semantically void reasoning on beyond-capability tasks.<n>We introduce textbfCaRL (textbfCapability-textbfaligned textbfReinforcement textbfL) to align behavior with capability boundaries.<n>Experiments demonstrate a substantial reduction in futile reasoning while preserving performance across task difficulties.
  arXiv  Detail & Related papers  (2026-07-31T09:30:33Z)
• Does Reasoning Preserve Alignment? On the Trustworthiness of Large Reasoning Models [55.788110316999166]
  We compare reasoning models produced via supervised fine-tuning, RL-based post-training, and distillation against instruction-tuned baselines.<n>We observe that reasoning models often improve on reasoning benchmarks but exhibit alignment regressions.<n>These regressions are consistent with behavioral drift from the instruction-tuned baseline, measured by KL divergence.
  arXiv  Detail & Related papers  (2026-06-09T16:14:27Z)
• Training on Documents About Monitoring Leads to CoT Obfuscation [4.971619873738039]
  Chain-of-thought (CoT) monitoring is one of the most promising tools for detecting model misbehavior.<n>We study whether monitor-aware models are capable of obfuscating their reasoning to evade detection.
  arXiv  Detail & Related papers  (2026-05-14T17:59:01Z)
• Correct Answers from Sound Reasoning: Verifiable Process Supervision for Language Models [94.68358825189738]
  We propose a post-training framework for verifiable domains that jointly optimize prediction accuracy and reasoning quality.<n>We evaluate VPS on chess, a controlled testbed where reasoning steps can be deterministically verified against engine signals.<n>VPS preserves accuracy while significantly improving reasoning quality, reducing win-rate error by up to 30% and restoring consistency to near saturation.
  arXiv  Detail & Related papers  (2026-04-03T15:19:46Z)
• Beyond Content Safety: Real-Time Monitoring for Reasoning Vulnerabilities in Large Language Models [14.927545906619295]
  Large language models increasingly rely on explicit chain-of-thought (CoT) reasoning to solve complex tasks.<n>Existing work on LLM safety focuses on content safety--detecting harmful, biased, or factually incorrect outputs.<n>We identify reasoning safety as a security dimension: the requirement that a model's reasoning trajectory be logically consistent.
  arXiv  Detail & Related papers  (2026-03-26T13:08:56Z)
• Noticing the Watcher: LLM Agents Can Infer CoT Monitoring from Blocking Feedback [48.40096834514452]
  Chain-of-thought (CoT) monitoring is proposed as a method for overseeing the internal reasoning of language-model agents.<n>We ask whether reasoning agents can autonomously infer that their supposedly private CoT is under surveillance.<n>We find that frontier models can deduce the existence of this monitor purely from blocking feedback.
  arXiv  Detail & Related papers  (2026-03-14T00:12:14Z)
• Controllable Reasoning Models Are Private Thinkers [74.40231123523115]
  We propose training models to follow instructions not only in the final answer, but also in reasoning traces.<n>We fine-tune models on an instruction-following dataset with explicit restrictions on reasoning traces.<n>Our results show that improving instruction-following behavior in reasoning models can significantly enhance privacy.
  arXiv  Detail & Related papers  (2026-02-27T17:39:10Z)
• Reasoning Under Pressure: How do Training Incentives Influence Chain-of-Thought Monitorability? [7.914706904029561]
  We investigate how different emphtraining incentives, applied to a reasoning model, affect its monitorability.<n>We find that adversarial optimisation (penalising monitor accuracy) degrades monitor performance, while direct optimisation for monitorability does not reliably lead to improvements.
  arXiv  Detail & Related papers  (2025-11-28T21:34:34Z)
• The Ends Justify the Thoughts: RL-Induced Motivated Reasoning in LLMs [2.583082967853897]
  We find that motivated reasoning can be detected by most frontier reasoning models.<n>We find that as models become more sophisticated, their motivated reasoning may become increasingly difficult for monitors to detect.
  arXiv  Detail & Related papers  (2025-10-20T00:24:08Z)
• Learning a Dense Reasoning Reward Model from Expert Demonstration via Inverse Reinforcement Learning [50.20267980386502]
  We learn a dense, token-level reward model for process supervision directly from expert demonstrations.<n>The learned reasoning reward serves two complementary roles: (i) it provides step-level feedback to optimise a reasoning policy during training; and (ii) it functions at inference as a critic to rerank sampled traces under fixed compute budgets.
  arXiv  Detail & Related papers  (2025-10-02T09:55:26Z)
• Beyond Linear Probes: Dynamic Safety Monitoring for Language Models [67.15793594651609]
  Traditional safety monitors require the same amount of compute for every query.<n>We introduce Truncated Polynomials (TPCs), a natural extension of linear probes for dynamic activation monitoring.<n>Our key insight is that TPCs can be trained and evaluated progressively, term-by-term.
  arXiv  Detail & Related papers  (2025-09-30T13:32:59Z)
• CoT Red-Handed: Stress Testing Chain-of-Thought Monitoring [3.6284577335311563]
  Chain-of-Thought (CoT) monitoring improves detection by up to 27 percentage points in scenarios where action-only monitoring fails to reliably identify sabotage.<n>CoT traces can also contain misleading rationalizations that deceive the monitor, reducing performance in more obvious sabotage cases.<n>This hybrid monitor consistently outperforms both CoT and action-only monitors across all tested models and tasks, with detection rates over four times higher than action-only monitoring for subtle deception scenarios.
  arXiv  Detail & Related papers  (2025-05-29T15:47:36Z)
• Mitigating Deceptive Alignment via Self-Monitoring [15.365589693661823]
  We develop a framework that embeds a Self-Monitor inside the chain-of-thought process itself, named CoT Monitor+.<n>During generation, the model produces (i) ordinary reasoning steps and (ii) an internal self-evaluation signal trained to flag and suppress misaligned strategies.<n>The signal is used as an auxiliary reward in reinforcement learning, creating a feedback loop that rewards honest reasoning and discourages hidden goals.
  arXiv  Detail & Related papers  (2025-05-24T17:41:47Z)
• Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation [56.102976602468615]
  We show that we can monitor a frontier reasoning model, such as OpenAI o3-mini, for reward hacking in agentic coding environments.<n>We find that with too much optimization, agents learn obfuscated reward hacking, hiding their intent within the chain-of-thought.
  arXiv  Detail & Related papers  (2025-03-14T23:50:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.