Related papers: Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety

Related papers

Trust as Monitoring: Evolutionary Dynamics of User Trust and AI Developer Behaviour [59.27481013639016]
We study how user trust strategies and developer choices between safe and unsafe AI co-evolve under different levels of monitoring cost and institutional regimes.<n>We find three robust long-run regimes: no adoption with unsafe development, unsafe but widely adopted systems, and safe systems that are widely adopted.
arXiv Detail & Related papers (2026-03-25T19:04:23Z)
Noticing the Watcher: LLM Agents Can Infer CoT Monitoring from Blocking Feedback [48.40096834514452]
Chain-of-thought (CoT) monitoring is proposed as a method for overseeing the internal reasoning of language-model agents.<n>We ask whether reasoning agents can autonomously infer that their supposedly private CoT is under surveillance.<n>We find that frontier models can deduce the existence of this monitor purely from blocking feedback.
arXiv Detail & Related papers (2026-03-14T00:12:14Z)
When can we trust untrusted monitoring? A safety case sketch across collusion strategies [43.912646993501106]
We develop methods for rigorously demonstrating safety based on pre-deployment testing.<n>We develop a taxonomy covering passive self-recognition, causal collusion (hiding pre-shared signals), acausal collusion (hiding signals via Schelling points) and combined strategies.
arXiv Detail & Related papers (2026-02-24T07:23:21Z)
Frontier AI Auditing: Toward Rigorous Third-Party Assessment of Safety and Security Practices at Leading AI Companies [57.521647436515785]
We define frontier AI auditing as rigorous third-party verification of frontier AI developers' safety and security claims.<n>We introduce AI Assurance Levels (AAL-1 to AAL-4), ranging from time-bounded system audits to continuous, deception-resilient verification.
arXiv Detail & Related papers (2026-01-16T18:44:09Z)
A Pragmatic Way to Measure Chain-of-Thought Monitorability [10.811252340660907]
Chain-of-Thought (CoT) monitoring offers a unique opportunity for AI safety.<n>To help preserve monitorability, we propose a pragmatic way to measure two components of it: legibility and coverage.<n>We implement these metrics with an autorater prompt that enables any capable LLM to compute the legibility and coverage of existing CoTs.
arXiv Detail & Related papers (2025-10-28T00:44:25Z)
A Concrete Roadmap towards Safety Cases based on Chain-of-Thought Monitoring [0.826731104724488]
This paper presents a roadmap for constructing safety cases based on chain-of-thought (CoT) monitoring in reasoning models.<n>We argue that CoT monitoring might support both control and trustworthiness safety cases.
arXiv Detail & Related papers (2025-10-22T11:13:52Z)
LLMs Can Covertly Sandbag on Capability Evaluations Against Chain-of-Thought Monitoring [5.214050557192032]
sandbagging is the strategic underperformance on evaluations by AI models or their developers.<n>One promising defense is to monitor a model's chain-of-thought (CoT) reasoning.<n>We show that both frontier models and small open-sourced models can covertly sandbag against CoT monitoring 0-shot without hints.
arXiv Detail & Related papers (2025-07-31T15:19:30Z)
Thought Purity: Defense Paradigm For Chain-of-Thought Attack [14.92561128881555]
We propose Thought Purity, a defense paradigm that strengthens resistance to malicious content while preserving operational efficacy.<n>Our approach establishes the first comprehensive defense mechanism against CoTA vulnerabilities in reinforcement learning-aligned reasoning systems.
arXiv Detail & Related papers (2025-07-16T15:09:13Z)
When Chain of Thought is Necessary, Language Models Struggle to Evade Monitors [10.705880888253501]
Chain-of-thought (CoT) monitoring is an appealing AI safety defense.<n>Recent work on "unfaithfulness" has cast doubt on its reliability.<n>We argue the key property is not faithfulness but monitorability.
arXiv Detail & Related papers (2025-07-07T17:54:52Z)
SafeCoT: Improving VLM Safety with Minimal Reasoning [5.452721786714111]
We introduce SafeCoT, a lightweight, interpretable framework to improve refusal behavior in vision-language models.<n>We show that SafeCoT significantly reduces overrefusal and enhances generalization, even with limited training data.
arXiv Detail & Related papers (2025-06-10T03:13:50Z)
Towards provable probabilistic safety for scalable embodied AI systems [79.31011047593492]
Embodied AI systems are increasingly prevalent across various applications.<n> Ensuring their safety in complex operating environments remains a major challenge.<n>We introduce provable probabilistic safety, which aims to ensure that the residual risk of large-scale deployment remains below a predefined threshold.
arXiv Detail & Related papers (2025-06-05T15:46:25Z)
An Approach to Technical AGI Safety and Security [72.83728459135101]
We develop an approach to address the risk of harms consequential enough to significantly harm humanity.<n>We focus on technical approaches to misuse and misalignment.<n>We briefly outline how these ingredients could be combined to produce safety cases for AGI systems.
arXiv Detail & Related papers (2025-04-02T15:59:31Z)
Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation [56.102976602468615]
We show that we can monitor a frontier reasoning model, such as OpenAI o3-mini, for reward hacking in agentic coding environments.<n>We find that with too much optimization, agents learn obfuscated reward hacking, hiding their intent within the chain-of-thought.
arXiv Detail & Related papers (2025-03-14T23:50:34Z)
To Think or Not to Think: Exploring the Unthinking Vulnerability in Large Reasoning Models [56.19026073319406]
Large Reasoning Models (LRMs) are designed to solve complex tasks by generating explicit reasoning traces before producing final answers.<n>We reveal a critical vulnerability in LRMs -- termed Unthinking -- wherein the thinking process can be bypassed by manipulating special tokens.<n>In this paper, we investigate this vulnerability from both malicious and beneficial perspectives.
arXiv Detail & Related papers (2025-02-16T10:45:56Z)
Safeguarded Progress in Reinforcement Learning: Safe Bayesian Exploration for Control Policy Synthesis [63.532413807686524]
This paper addresses the problem of maintaining safety during training in Reinforcement Learning (RL) We propose a new architecture that handles the trade-off between efficient progress and safety during exploration.
arXiv Detail & Related papers (2023-12-18T16:09:43Z)
SoK: Modeling Explainability in Security Analytics for Interpretability, Trustworthiness, and Usability [2.656910687062026]
Interpretability, trustworthiness, and usability are key considerations in high-stake security applications. Deep learning models behave as black boxes in which identifying important features and factors that led to a classification or a prediction is difficult. Most explanation methods provide inconsistent explanations, have low fidelity, and are susceptible to adversarial manipulation.
arXiv Detail & Related papers (2022-10-31T15:01:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.