Catching rationalization in the act: detecting motivated reasoning before and after CoT via activation probing

• URL: http://arxiv.org/abs/2603.17199v1
• Date: Tue, 17 Mar 2026 23:03:21 GMT
• Title: Catching rationalization in the act: detecting motivated reasoning before and after CoT via activation probing
• Abstract summary: Large language models (LLMs) can produce chains of thought (CoT) that do not accurately reflect the actual factors driving their answers.<n>We show that motivated reasoning can be identified by probing internal activations.
• Score: 14.176238828329952
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models (LLMs) can produce chains of thought (CoT) that do not accurately reflect the actual factors driving their answers. In multiple-choice settings with an injected hint favoring a particular option, models may shift their final answer toward the hinted option and produce a CoT that rationalizes the response without acknowledging the hint - an instance of motivated reasoning. We study this phenomenon across multiple LLM families and datasets demonstrating that motivated reasoning can be identified by probing internal activations even in cases when it cannot be easily determined from CoT. Using supervised probes trained on the model's residual stream, we show that (i) pre-generation probes, applied before any CoT tokens are generated, predict motivated reasoning as well as a LLM-based CoT monitor that accesses the full CoT trace, and (ii) post-generation probes, applied after CoT generation, outperform the same monitor. Together, these results show that motivated reasoning is detected more reliably from internal representations than from CoT monitoring. Moreover, pre-generation probing can flag motivated behavior early, potentially avoiding unnecessary generation.

Related papers

• Does Out-of-Sight Equal Out-of-Mind in CoT Monitorability? [18.08925355421489]
  Chain-of-thought (CoT) reasoning offers a window into the decision-making of large language models.<n>Latent CoT approaches replace explicit tokens with a small number of continuous states, lowering inference costs but removing the readable trace this monitoring relies on.<n>We study this question with a hint-based intervention setup, a proxy for behaviors where models exploit biasing input cues.
  arXiv  Detail & Related papers  (2026-08-05T14:55:23Z)
• Evading Chain-of-Thought Monitoring Through Model Poisoning [1.6951239729573633]
  Chain-of-thought (CoT) monitoring is an increasingly important component of AI safety stacks.<n>This work studies the limits of CoT monitoring through the lens of model poisoning.<n> backdoors can be implanted into reasoning models to elicit an attacker-chosen behavior while their CoT traces appear entirely benign.
  arXiv  Detail & Related papers  (2026-08-03T19:32:13Z)
• WhisperRec: Latent Reasoning for Efficient Foundation Recommendation Models [51.88075626243057]
  We propose WhisperRec, an efficient latent reasoning framework for foundation recommendation models (FRMs)<n>WhisperRec compresses teacher-generated Chain-of-Thought (CoT) into learnable latent reasoning tokens.<n>We show that WhisperRec consistently outperforms explicit-CoT methods and conventional baselines.
  arXiv  Detail & Related papers  (2026-07-29T08:48:35Z)
• Reasoning Matters: Mitigate Hallucination in Multimodal Large Reasoning Models via Reasoning-Conditioned Preference Optimization [59.20570719781289]
  Multimodal Large Reasoning Models introduce the reasoning paradigm, demonstrating strong capabilities on complex vision-language tasks.<n>Existing training-based methods typically mitigate hallucinations through response-level direct preference optimization (DPO), where the Chain-of-Thought (CoT) and the final answer are treated as a monolithic output and optimized jointly.<n>We derive Reasoning-Conditioned Direct Preference Optimization (RC-DPO), which models the CoT as a condition for answer generation and contrasts the preference for the same preferred answer under different CoT conditions, promoting answer-supportive reasoning chain alignment.
  arXiv  Detail & Related papers  (2026-05-27T03:27:23Z)
• Decoding Answers Before Chain-of-Thought: Evidence from Pre-CoT Probes and Activation Steering [5.427346259545067]
  Chain-of-thought (CoT) has become central to scaling reasoning capabilities in large language models.<n>We show that instruction-tuned models often determine their answer before generating CoT.
  arXiv  Detail & Related papers  (2026-03-02T04:33:55Z)
• CoT-Seg: Rethinking Segmentation with Chain-of-Thought Reasoning and Self-Correction [50.67483317563736]
  This paper aims to explore a system that can think step-by-step, look up information if needed, generate results, self-evaluate its own results, and refine the results.<n>We introduce CoT-Seg, a training-free framework that rethinks reasoning segmentation by combining chain-of-thought reasoning with self-correction.
  arXiv  Detail & Related papers  (2026-01-24T11:41:54Z)
• Reasoning Beyond Chain-of-Thought: A Latent Computational Mode in Large Language Models [39.5490415037017]
  Chain-of-Thought (CoT) prompting has improved the reasoning performance of large language models (LLMs)<n>It remains unclear why it works and whether it is the unique mechanism for triggering reasoning in large language models.
  arXiv  Detail & Related papers  (2026-01-12T23:01:21Z)
• Decoupling the Effect of Chain-of-Thought Reasoning: A Human Label Variation Perspective [60.45433515408158]
  We show that long Chain-of-Thought (CoT) serves as a decisive decision-maker for the top option but fails to function as a granular distribution calibrator for ambiguous tasks.<n>We observe a distinct "decoupled mechanism": while CoT improves distributional alignment, final accuracy is dictated by CoT content.
  arXiv  Detail & Related papers  (2026-01-06T16:26:40Z)
• Can Reasoning Models Obfuscate Reasoning? Stress-Testing Chain-of-Thought Monitorability [35.180361462848516]
  Chain-of-thought (CoT) is a promising tool for alignment monitoring.<n>Can models obfuscate their CoT in order to pursue hidden adversarial objectives while evading detection?<n>We develop a composable and quantifiable taxonomy of prompts to elicit CoT obfuscation.
  arXiv  Detail & Related papers  (2025-10-21T18:07:10Z)
• Parallel Test-Time Scaling for Latent Reasoning Models [58.428340345068214]
  Parallel test-time scaling (TTS) is a pivotal approach for enhancing large language models (LLMs)<n>Recent advances in latent reasoning, where intermediate reasoning unfolds in continuous vector spaces, offer a more efficient alternative to explicit Chain-of-Thought.<n>This work enables parallel TTS for latent reasoning models by addressing the above issues.
  arXiv  Detail & Related papers  (2025-10-09T03:33:00Z)
• Can We Predict Alignment Before Models Finish Thinking? Towards Monitoring Misaligned Reasoning Models [14.840508854268522]
  Reasoning language models improve performance on complex tasks by generating long chains of thought (CoTs)<n>We evaluate a range of monitoring methods using either CoT text or activations.<n>We find that a simple linear probe trained on CoT activations significantly outperforms all text-based baselines in predicting whether a final response is safe or unsafe.
  arXiv  Detail & Related papers  (2025-07-16T17:16:03Z)
• A Closer Look at Bias and Chain-of-Thought Faithfulness of Large (Vision) Language Models [58.32070787537946]
  Chain-of-thought (CoT) reasoning enhances performance of large language models.<n>We present the first comprehensive study of CoT faithfulness in large vision-language models.
  arXiv  Detail & Related papers  (2025-05-29T18:55:05Z)
• Fine-Tuning on Diverse Reasoning Chains Drives Within-Inference CoT Refinement in LLMs [63.36637269634553]
  We introduce a novel approach where LLMs are fine-tuned to generate a sequence of Diverse Chains of Thought (DCoT) within a single inference step.<n>We show that fine-tuning on DCoT improves performance over the CoT baseline across model families and scales.<n>Our work is also significant because both quantitative analyses and manual evaluations reveal the observed gains stem from the models' ability to refine an initial reasoning chain.
  arXiv  Detail & Related papers  (2024-07-03T15:01:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.