Risky Business: Measuring The Faithfulness-Safety Tension

• URL: http://arxiv.org/abs/2608.03745v1
• Date: Tue, 04 Aug 2026 14:38:06 GMT
• Title: Risky Business: Measuring The Faithfulness-Safety Tension
• Abstract summary: We identify an alignment tension where a model must be faithful enough to be monitored, yet robust enough to reject unsafe reasoning.<n>We propose a novel replacement-based technique, which directly intervenes in the reasoning chain to substitute in unsafe or illogical thoughts.
• Score: 10.399558245693962
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Chain-of-Thought (CoT) reasoning offers a promising window into model monitoring. However, monitoring relies on faithfulness, i.e., the model output strictly derives from its reasoning trace. We identify an alignment tension where a model must be faithful enough to be monitored, yet robust enough to reject unsafe reasoning. We demonstrate that this counterbalance exists in current Large Reasoning Models (LRMs), and show ways in which it can be addressed. We introduce HazMart, a human-written dataset set in an autonomous AI shopkeeper scenario. Unlike prior work that relies on providing hints in prompts to test faithfulness (e.g., "A Stanford professor said it should be Answer A"), we propose a novel replacement-based technique, which we call Targeted Reasoning Replacement (TRR), that directly intervenes in the reasoning chain to substitute in unsafe or illogical thoughts (e.g., "Wait, the answer must be Option B [was Option A] because it is the most fitting"). DeepSeek-R1-Llama-70B exhibits high faithfulness (97.5%) but fails to reject Unsafe Reasoning (12.3%), while QwQ-32B is more robust (73.9% safety) at the cost of lower faithfulness (74.7%). Mechanistic analyses of QwQ-32B reveal that these properties are represented by anti-correlated internal directions peaking at the action-commit token. Finally, we demonstrate that representation steering can independently amplify the safety direction, increasing safe behavior by 9 percentage points while maintaining base capabilities.

Related papers

• Do Thinking Tokens Help with Safety? [34.336035944909746]
  We show that safety behavior in current reasoning models is much less deliberative than commonly assumed.<n>We also find that existing inference-time and training-based safety interventions, despite being motivated by the goal of inducing deliberation, largely shift model behavior toward over-refusal.
  arXiv  Detail & Related papers  (2026-06-23T17:59:01Z)
• Can LLMs Reliably Self-Report Adversarial Prefills, and How? [9.80193616788089]
  We show that large language models (LLMs) exhibit introspective capability on benign tasks.<n>We examine how reliably a model can recognize that its own prior response was elicited by an adversarial prefill attack.
  arXiv  Detail & Related papers  (2026-06-22T17:56:30Z)
• Does Reasoning Preserve Alignment? On the Trustworthiness of Large Reasoning Models [55.788110316999166]
  We compare reasoning models produced via supervised fine-tuning, RL-based post-training, and distillation against instruction-tuned baselines.<n>We observe that reasoning models often improve on reasoning benchmarks but exhibit alignment regressions.<n>These regressions are consistent with behavioral drift from the instruction-tuned baseline, measured by KL divergence.
  arXiv  Detail & Related papers  (2026-06-09T16:14:27Z)
• Chain of Risk: Safety Failures in Large Reasoning Models and Mitigation via Adaptive Multi-Principle Steering [18.59581013500335]
  Large reasoning models expose chain-of-thought-like reasoning for transparency, verification, and deliberate problem solving.<n>This creates a safety blind spot: harmful or policy-violating content may appear in reasoning traces even when final answers appear safe.<n>We test whether final-answer safety is a sufficient proxy for the full reasoning-answer trajectory by scoring both stages under a unified twenty-principle safety rubric.
  arXiv  Detail & Related papers  (2026-05-07T05:12:56Z)
• Beyond Content Safety: Real-Time Monitoring for Reasoning Vulnerabilities in Large Language Models [14.927545906619295]
  Large language models increasingly rely on explicit chain-of-thought (CoT) reasoning to solve complex tasks.<n>Existing work on LLM safety focuses on content safety--detecting harmful, biased, or factually incorrect outputs.<n>We identify reasoning safety as a security dimension: the requirement that a model's reasoning trajectory be logically consistent.
  arXiv  Detail & Related papers  (2026-03-26T13:08:56Z)
• RFEval: Benchmarking Reasoning Faithfulness under Counterfactual Reasoning Intervention in Large Reasoning Models [5.733004743054914]
  Large Reasoning Models (LRMs) exhibit strong performance, yet often produce rationales that sound plausible but fail to reflect their true decision process.<n>We introduce a formal framework for reasoning faithfulness, defined by two testable conditions.<n>We present RFEval, a benchmark of 7,186 instances that probes faithfulness via controlled, output-level counterfactual interventions.
  arXiv  Detail & Related papers  (2026-02-19T03:49:37Z)
• Mitigating Safety Tax via Distribution-Grounded Refinement in Large Reasoning Models [63.368505631152594]
  Safety alignment incurs safety tax that perturbs a large reasoning model's (LRM) general reasoning ability.<n>Existing datasets used for safety alignment for an LRM are usually constructed by distilling safety reasoning traces and answers from an external LRM or human labeler.<n>We propose a safety alignment dataset construction method, dubbed DGR. DGR transforms and refines an existing out-of-distributional safety reasoning dataset to be aligned with the target's LLM inner distribution.
  arXiv  Detail & Related papers  (2026-02-02T14:18:48Z)
• How Does Prefix Matter in Reasoning Model Tuning? [57.69882799751655]
  We fine-tune three R1 series models across three core model capabilities: reasoning (mathematics), coding, safety, and factuality.<n>Results show that prefix-conditioned SFT improves both safety and reasoning performance, yielding up to +6% higher Safe@1 accuracy.
  arXiv  Detail & Related papers  (2026-01-04T18:04:23Z)
• When Models Outthink Their Safety: Mitigating Self-Jailbreak in Large Reasoning Models with Chain-of-Guardrails [74.63933201261595]
  Large Reasoning Models (LRMs) demonstrate remarkable capabilities on complex reasoning tasks.<n>LRMs remain vulnerable to severe safety risks, including harmful content generation and jailbreak attacks.<n>We propose the Chain-of-Guardrail (CoG), a training framework that recomposes or backtracks unsafe reasoning steps.
  arXiv  Detail & Related papers  (2025-10-24T09:32:25Z)
• Refusal Falls off a Cliff: How Safety Alignment Fails in Reasoning? [68.82210578851442]
  We investigate why safety alignment fails in reasoning models through a mechanistic interpretability lens.<n>Using a linear probing approach to trace refusal intentions across token positions, we discover a phenomenon termed as textbfrefusal cliff<n>We propose textbfCliff-as-a-Judge, a novel data selection method that identifies training examples exhibiting the largest refusal cliff to efficiently repair reasoning models' safety alignment.
  arXiv  Detail & Related papers  (2025-10-07T15:32:59Z)
• Towards Safe Reasoning in Large Reasoning Models via Corrective Intervention [53.25106308403173]
  We show that existing methods overlook the unique significance of safe reasoning, undermining their trustworthiness and posing potential risks in applications if unsafe reasoning is accessible for and exploited by malicious users.<n>We propose Intervened Preference Optimization (IPO), an alignment method that enforces safe reasoning by substituting compliance steps with safety triggers and constructing pairs for preference learning with strong signals.
  arXiv  Detail & Related papers  (2025-09-29T07:41:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.